You know what? We’ve been a little system heavy the last few months. Lets try something different…
So, we all know JIRA workflows. They guide your issues through their life-cycle, apparently only know three colors, and otherwise just sit there, right? What if I told you that you can train your workflows to do tricks? It’s true!
Today we will go over some of the ways I have trained my workflows over the years, and when you’d want to use each trick. The ultimate goal here is to get you to see your workflow as an equal part of the issue life-cycle – that is to say not only as a faithful guide, but an active participant!
Some of these tricks will require a plugin to enable – but where so I will clearly mark them as such and give you a link to the plugin.
With that being said, let’s get into it.
Conditions, Validators, and Post Functions, Oh My!
So to get started, we’ll need to review the three parts of a transition: The Condition, Validator, and Post Function. Each of these are evaluated at a different part of the Transition execution, and therefore are able to do different things.
Conditions
These are evaluated any time JIRA needs to see if you are even able to see that the transitions exists. On the surface, seems straight forward enough. If a user doesn’t meet the conditions, they don’t see the transition, so what tricks could we do with that?
Trick: Secret Cleanup Transition
So, here’s the situation. As a part of your workflow, you require certain information be filled in to close issues normally. For example, you require a fix version be assigned to a new feature or bug. This is because your users had a bad habit of just closing critical bugs that really needed to be fixed.
Which is great, but as time goes along, the project lead comes to you, saying they have too many issues that being honest, they won’t ever get to. However, you realize that because of the requirements, you can’t just close those issues. You obviously don’t want to get rid of the requirements, but you want to enable the Project lead to do their cleanup. How do you do this?
We can do this simply with one condition! First we’ll add an “All” transition to the closed state we’ll call “Cleanup” Then we’ll put a condition on it that says only Project Administrators can see this cleanup transition. This way we don’t open up the workflow to the abuse we’ve previously tried to stop, but we give the Project lead the power to cleanup when they need to.
Validators
Validators are similar to conditions in the fact that they evaluate whether a transition could occur. However, the key different is WHEN they do their processing. Conditions are evaluated before a transition ever occurs. On the other hand, a validator is evaluated during the processing of the transition.
One thing Validators are great for is that when they fail, they provide an error message. When a condition fails, the relevant transition just doesn’t show up. From experience, this usually leads to a lot of panicked questions from new people saying “I can’t close an issue!” This problem is avoided entirely if you use a validator instead. “Oh, I can’t close it because I need a Fix Version – cool.”
So, you have an approval chain workflow, but you want to be sure whoever approved it really MEANT it. Lets say this involves a P.O. process, so you want to know whoever approves it really meant you to spend that money.
This is easy enough with JMWE’s “Comment Required” Validator. To make sure the phrase is added as, we’ll add a conditional to evaluate the comment field.
issue.getAsString("comment") != "I Approve"
This is a bit counter-intuitive, but the condition is what makes the validator active in JWME. Here, the validator only ever validates if the comment is anything other than “I Approve” – which includes an empty string! Having the comment be “I Approve” disables the validator and lets the transition occur!
Post Functions
Post functions have no evaluation purpose, but are executed “After” all conditions are evaluated and things are ready to move forward. This gives you a chance to update fields, fire off internal events or web-hooks, or a myriad of other tasks. Most of the time when I’m looking to automate some function that is based on a workflow, Post functions are where I’ll look to first.
Trick: Auto-assign on transition
So this is great for any situations where a specific person needs to do something on a specific status….such as an approval chain. This lets you set it up to reassign the issue on demand without having the user remember to take that action. Very Handy!
To accomplish this, we’ll use the “Update Issue Field” Post function, select “Assignee” from the drop-down, then select the last radio button to specify which user you’d like to transfer it to.
There are additioonal post functions from various Apps that will do the same thing, but you can do this with just vanilla JIRA Functionality.
Workflow Properties: The forbidden technique
Okay, so maybe I’m being a little dramatic…
Transition and Status properties are a powerful technique for customizing how your workflow behaves. However, they are somewhat hidden, dubiously supported, and tricky to get right. Basically, if you do choose to go this route, you will be pretty much on your own, with your only company being whatever articles you can google. By the way, Hi googlers! Welcome to the blog!
In general, I don’t really like to use Workflow properties, but there are some cases where they can’t be helped. One use case I’ve used them for is to limit who can reassign an issue in a given status…which being honest I learned from another blog.
To access the properties screen, select your transition or status of your choice, then click “Properties”
This will open up another screen where you can see a list of that item’s properties. By default there should be none.
Trick: Re-order the Transition buttons
So lets say you have a project lead that is a little…particular. They want the most relevant transition to be the first button, with anything else being second. But you can’t re-order the Transition buttons, right?
Well, I wouldn’t be writing this section if that was the case. We can use a property on each transition called opsbar-sequence. This is a setting that will allow us to specify which transition gets priority on the issue view screen. To do this, we’ll need to add the key “opsbar-sequence” to each transition, along with a unique value. It is recommend you increment the value by 10’s on each transitiono, so if you add more transitions later, you have a place to put them between existing transitions
Remember, in this arrangement, the lowest opsbar-sequence value will go first, with each increasing value following afterwards.
You are now a workflow trainer!
My goal here is to get you to think about different automations you can enable for your users. It may seem trivial to you, but if you can save them 10 seconds on each transaction, that can really add up over time. And you don’t have to stop with this list! Look over what post functions, validators, conditions, and yes even properties you have available, and keep an open mind when you get these unusual requests.
However, a word of caution. If you go crazy with these, you may find you have too many workflows and actions within workflows to manage. It’s best to think of it like candy. Every once in a while is fine, and can make you feel happy for a bit. But too much too often can make you regret it before too long!
As always, don’t forget to subscribe to be notified by email of new posts! You’ll find a signup form at the bottom of the post. Also don’t forget to check out our Atlassian Discord Chat! It’s still ramping up, but I’m always happy to see people getting questions answered there! https://discord.gg/mXuRsVu
But until next time, this is Rodney, asking “Have you updated your JIRA issues today?”
So it’s that time again. And this time only two weeks behind!
Atlassian released JIRA Core and Software 8.7, as well as JIRA Service Desk 4.7, a few weeks ago. Unlike JIRA 8.6, this only only has a few new features, with most of them relegated to JIRA Service Desk, but I figured I’d take a look at them all the same.
As I mentioned in my last release review, JIRA Server and Data Center are released in discrete releases. As an admin, you should always review each one. Even if you are staying on Enterprise Releases, it will give you ideas of what features to look forward to and promote when you do your next upgrade.
If this release has a theme, I feel it is Privacy and Security. Almost every new feature touches on one of these two ideas in one way or another. There is only a few new features in JIRA Software and Core, with Service Desk having more. So we’ll cover all the universal new features, then go over what’s new in Service Desk.
GDPR Anonymizing
I’ll admit, other than some audit headaches, GDPR hasn’t impacted me too much. However, I’m still a fan of the rules and are hoping something similar is adopted over here in the United States.
However, for those of you who are subject to GDPR Rules, Atlassian is helping you out. You can now anonymize any information tied to a user that can identify them upon request. This is a one way process, with no way to undo it, but that is kind of the point.
Courtesy Atlassian Knowledge Base
Included is a wizard that will search that user, show you any items in JRIA that stores that user’s personal information, then let you Anonymize them. Honestly, it looks great and it’s one less headache at my next audit, so I like this one!
PostgreSQL 11 Support
This one is big. As you may know, Atlassian products can run on a number of Database architectures. However, what you may not know is that they run best on PostgreSQL. Now I’m personally still partial to MySQL, but when it comes to getting every bit of performance I can out of these applications, knowing that I have this as an option is always nice.
However, until JIRA 8.7, the most recent PostgreSQL supported was PostgreSQL 10, and the only PostgreSQL 9 with continuing support being 9.6. So it’s always nice to have them add another version in. Now here’s hoping they can get around to MySQL 8 before 5.7 is end of life’d.
OpenID Connect for Data Center
And a feature coming to Data Center instances with this release is OpenID Connect. This gives you another tool to connect with various Identity Providers. Another arrow in the quiver when trying to integrate JIRA is always appreciated.
However, for JIRA Core and Software, these are all the new features. However there was a number of bugs also fixed, so that might also be something you will look at when determining if a particular version is right for you.
New Service Desk Features
Keeping your requests Private
The first new Service Desk feature is really a change to a default option. As of Service Desk 4.7, any request made through the Customer portal is Private by default. You can choose to share them with people in your organization, but if you don’t change any settings, it will be private.
Image courtesy of Atlassian Knowledge Base
Requests originating from an email still is shared by default, but you as an admin now have the ability to change the default behavior within Administration > Applications > Jira Service Desk configuration.
Verifying Customer’s Email
For those of you who have enabled public signup through your customer portal, you can also add a step to have them verify their email address. This should make it easier to get in touch with your customers, as you’ll know that the email they signed up with actually belongs to them.
Agents are well…agents
This one has aggravated me. When commenting on issues or acting as an approver, your agents were treated as customers by JIRA. This would muck up all sorts of things, from SLA’s to automation – which was my headache. It could even confuse customers – which is the last thing you want. So they are changing it so agents are well…agents! Novel concept I know.
So….should I upgrade to this?
Well….That’s a whole big bag of “It Depends.” If you are not currently on an Enterprise release, it would be tempting to just get the latest bug fixes. But ultimately I think this question comes down to is the question, “Do I need the new features?” Some of you may see the GDPR features and go “Yes”. Others will be able to wait for the next Enterprise release. Ultimately it is up to you to look at the features and see if the value justifies the time it will take to do an upgrade cycle.
And that’s it for this week!
A bit of a shorter article this week, but it’s been a busy week. My wife and I are getting ready for a long weekend trip together, and I’ve still got a lot of getting ready to do.
As I told you guys last week, I recently helped out with the Q & A section of a webinar my company put on. Well, I also helped out on the write-up of the questions and their answers afterwards. So if you have some free time and want some good answers to questions people had about Atlassian Cloud, please check it out!
Don’t forget to subscribe below to get notified about new blog posts, and also about our Atlassian Discord Chat. While unofficial, it’s a good place to get help, discuss things, and connect with your fellow Atlassian Admins. https://discord.gg/mXuRsVu
But until next time, my name is Rodney, asking “Have you updated your JIRA issues today?”
So, dear readers, here’s the deal. Some weeks, when I sit down to write, I know exactly what I’m going to write about, and can get right to it. Other weeks, I’m sitting down, and I don’t have a clue. I can usually figure something out, but it’s very much a struggle. This week is VERY much the latter.
Compound that with the fact that I just lost most of my VM’s due to a storage failure I had this very morning. Part of it was a mistake on my part. I have the home lab so that I can learn things I can’t learn on the job. And mistakes are a painful but powerful way to learn. Still….
This brings me back to a conversation I had with a colleague and fellow Atlassian Administrator for a company I used to work for. He had asked me what my thoughts around implementing Monitoring of JIRA. Well, I have touched on the subject before, but if I’m being honest, this isn’t my greatest work. Combine that with the fact that I suddenly need to rebuild EVERYTHING, well, why not start with my monitoring stack!
So, we are going to be setting up a number of systems. To gather system stats, that is to say CPU usage, Memory Usage, and Disk usage, we are going to be using Telegraf, which will be storing that data in an InfluxDB database. Then for JIRA stats we are going to use Prometheus. And to query and display this information, we will be using Grafana.
The Setup
So we are going to be setting up a new system that will live alongside our JIRA instance. We will call it Grafana, as that will be the front end we will interact with the system with.
On the back end it will be running both a InfluxDB Server and a Prometheus Server. Grafana will use both InfluxDB and Prometheus as data sources, and will use that to generate stats and graphs of all the relevant information.
Our system will be a CentOS 7 system (my favorite currently), and will have the following stats:
2 vCPU
4 GB RAM
16 GB Root HDD for OS
50 GB Secondary HDD for Services
This will give us the ability to scale up the capacity for services to store files without too much impact on the overall system, as well as monitor it’s size as well.
As per normal, I am going to write all commands out assuming you are root. If you are not, I’m also assuming you know what sudo is and how to use it, so I won’t insult you by holding your hand with that.
InfluxDB
Lets get started with InfluxDB. First thing we’ll need to do is add the yum repo from Influxdata onto the system. This will allow us to use yum to do the heavy lifting in the install of this service.
Now we need to set some credentials. As initially setup, the system isn’t really all that secure. So we are going to secure it initially by using curl to set ourselves an account.
curl -XPOST "http://localhost:8086/query" --data-urlencode \
"q=CREATE USER username WITH PASSWORD 'strongpassword' WITH ALL PRIVILEGES"
I shouldn’t have to say this, but you should replace username with one you can remember and strongpassword with, well, a strong password.
Now we can use the command “influx” to get into InfluxDB and do any further set up we need.
influx -username 'username' -password 'password'
Now that we are in, we need to setup a database and user for our JIRA data to go into. As a rule of thumb, I like to have one DB per application and/or system I intend to monitor with InfluxDB.
CREATE DATABASE Jira
CREATE USER jira WITH PASSWORD 'strongpassword'
GRANT ALL ON jira TO jira
CREATE RETENTION POLICY one_year ON Jira DURATION 365d REPLICATION 1 DEFAULT
SHOW RETENTION POLICIES ON Jira
And that’s it, InfluxDB is ready to go!
Grafana
Now that we have at least one datasource, we can get to setting up the Front End. Unfortunately, we’ll need information from JIRA in order to setup Prometheus (once we’ve set JIRA up to use the Prometheus Exporter), so that data source will need to wait.
Fortunately, Grafana can also be setup using a Yum repo. So lets open up /etc/yum.repos.d/grafana.repo
Go to port 3000 of the system on your web browser and you should see it up and running. We’ll hold off on setting up everything else on Grafana until we finish the system setup, though.
Telegraf
Telegraf is the tool we will use to get our data from JIRA’s underlying linux system and into InfluxDB. This is actually part of the same YUM repo that InfluxDB is installed from, so we’ll now also add it to the JIRA server – same as we did Grafana.
And now that it has the YUM repo, we’ll install telegraf onto the JIRA Server.
yum install telegraf -y
Now that we have it installed, we can take a look at it’s configuration, which you can find in /etc/telegraf/telegraf.conf. I highly suggest you take a backup of this file first. Here is an example of a config file where I’ve filtered out all the comments and added back in everything essential.
And that should be it for config. There are of course more we can capture using various plugins – based on whatever we are interested in, but this will get the bare minimum we are interested in.
Because telegraf is pushing data to the InfluxDB server, we don’t need to open any firewall ports for this, which means we can start it, then monitor the logs to make sure it is sending the data over without any problems.
And assuming you don’t see any errors here, you are good to go! We will have the stats waiting for us when you finish the setup of Grafana. But first….
Prometheus Exporter
So telegraf is great for getting the Linux system stats, but that only gives us a partial picture. We can train it to capture JMX info, but that means we have to setup JMX – something I’m keen to avoid whenever possible. So what options have we got to capture details like JIRA usage, JAVA Heap performance, etc?
That’s right, as of the time of this writing, this is yet another free app! This will setup a special page that Prometheus can go to and “scrape” the data from. This is what will take our monitoring from “okay” to “Woah”.
Because it is a free app, we can install it directly from the “Manage Apps” section of the JIRA Administration console
Once you click install, click “Accept & Install” on the pop up, and it’s done! After a refresh, you should notice a new sidebar item called “Prometheus Exporter Settings”. Click that, then click “Generate” next to the token field.
Next we’ll need to open the “here” link into a new tab on the “Exposed metrics are here” text. Take special special note of the URL used, as we’ll need this to setup Prometheus.
Prometheus
Now we’ll go back to our Grafana system to setup Prometheus. To find the download, we’ll go to the Prometheus Download Page, and find the latest Linux 64 bit version.
Try to avoid “Pre-release”
Copy that to your clipboard, then download it to your Grafana system.
Now if we go into the prometheus folder, we will see a normal assortment of files, but the one we are interested in is prometheus.yml. This is our config file and where we are interested in working. As always, take a backup of the original file, then open it with:
vim /archive/prometheus/prometheus.yml
Here we will be adding a new “job” to the bottom of the config. You can copy this config and modify it for your purposes. Note we are using the URL we got from the Prometheus Exporter. The first part of the URL (everything up to the first slash, or the FQDN) goes under target where indicated. The rest of the URL (folder path) goes under metrics_path. And then your token goes where indicated so that you can secure these metrics.
global:
scrape_interval: 15s
evaluation_interval: 15s
alerting:
alertmanagers:
- static_configs:
- targets:
rule_files:
scrape_configs:
- job_name: 'prometheus'
static_configs:
- targets: ['localhost:9090']
- job_name: 'Jira'
scheme: https
metrics_path: '<everything after the slash>'
params:
token: ['<token from Prometheus exporter']
static_configs:
- targets:
- <first part of JIRA URL, everything before the first '/'>
We’ll need to now open up the firewall port for Prometheus
Now we can test Prometheus. from the prometheus folder, run the following command.
./prometheus --config.file=prometheus.yml
From here we can open a web browser, and point it to our Grafana server on port 9090. On the Menu, we can go to Status -> Targets and see that both the local monitoring and JIRA are online.
Go ahead and stop prometheus for now by hitting “Ctrl + C”. We’ll need to set this up as a service so that we can rely on it coming up on it’s own should we ever have to restart the Grafana server.
Start by creating a unique user for this service. We’ll be using the options “–no-create-home” and “–shell /bin/false” to tell linux this is an account that shouldn’t be allowed to login to the server.
Now we’ll change the files to be owned by this new prometheus account. Note that the -R makes chown run recursively, meaning it will change it for every file underneath were we run it. Stop and make sure you are running it from the correct directory. If you run this command from the root directory, you will have a bad day (Trust me)!
After you save this file, type the following commands to reload systemctl, start the service, make sure it’s running, then enable it for launch on boot.:
Now just double check that the service is in fact running, and you’re good to go!
Grafana, the Reckoning
Now that we have both our datasources up and gathering information, we need to start by creating a way to display it. On your web browser, go back to Grafana, port 3000. You should be greeted with the same login screen as before. To login the first time, use ‘admin’ as username and password.
You will be prompted immediately to change this password. Do so. No – really.
After you change your password, you should see the screen below. Clilck “Add data source”
We’ll select InfluxDB from the list as our first Data Source.
Password: Whatever you set the InfluxDB Jira password to be
Click “Save & Test” at the bottom and you should have the first one down. Now click “Back” so we can set up Prometheus.
On Prometheus, all we’ll need to do is set the URL to be “http://localhost:9090. Enter that, then click “Save & Test”. And that’s both Data Sources done! Now we can move onto the Dashboard. On the right sidebar, click through to “Home”, then click “New Dashboard”
And now you are ready to start visualizing Data. I’ve already covered some Dashboard tricks in my previous attempt at this topic. However, if it helps, here’s how I used Prometheus to setup a graph of the JVM Heap.
Some Notes
Now, there is some cleanup you can do here. You can map out the storage for Grafana and InfluxDB to go to your /archive drive, for example. However, I can’t be giving away *ALL* the secrets ;). I want to challenge you there to see if you can learn to do it yourself.
We do have a few scaling options here too. For one, we can split Influx, Prometheus, and Grafana onto their own systems. However, my experience has been that this isn’t usually necessary, and they can all live comfortably on one system.
And one final note. The Prometheus exporter, strictly speaking, isn’t JIRA Data Center compatible. It will run however. As best I can tell, it will give you the stats for each node where applicable, and the overall stats where that makes sense. It might be worth installing and setting up Prometheus to bypass the load balancer and do each node individually.
But seriously, that’s it?
Indeed it is! This one is probably one of my longer posts, so thank you for making it to the end. It’s been a great week hearing how the blog is helping people out in their work, so keep it up! I’ll do my part here to keep providing you content.
On that note, this post was a reader-requested topic. I’m always happy to take on a challenge from readers, so if you have something you’d like to hear about, let me know!
One thing that I’m working on is to try and make it easier for you to be notified about new blog posts. As such, I’ve included an email subscription form at the bottom of the blog. If you want to be notified automatically about to blog posts, enter your email and hit subscribe!
And don’t forget about the Atlassian Discord chat – thoroughly unofficial. click here to join: https://discord.gg/mXuRsVu
But until next time, my name is Rodney, asking “Have you updated your JIRA issues today?”
As you know, I don’t normally like to post mid-week. I also try to keep the blog and my work-life separate (no small feat considering they are both based around supporting Atlassian systems). However, I thought you might be interested in this.
I have been asked to host the Q&A Section of a webinar my company is giving Wednesday. The topic is going to be “Is Your Organization Atlassian Cloud Ready?” If this sounds like something you’d be interested in please follow the link here to sign up.
I’ll have a post ready for you at our normal time this Wednesday, with the webinar following shortly afterwards! Until then, I’m Rodney, asking “Have you updated your JIRA issues today?”
So last week we took a dive into upgrading JIRA Server. So naturally this week it stands to reason that we will go over it’s big brother, JIRA Data Center. To be fair, doing an upgrade in Data Center isn’t much different than in Server. There are minor changes though that take full advantage of Data Center’s architecture, and those are what we’ll be focusing on today. We can sit here jabbering about it for a while, or we could get into it.
Zero Downtime Upgrades
So one of JIRA Data Center’s super-powers is the ability to stay online and completely functional while you are doing an upgrade! Called “Zero Downtime Upgrades” by Atlassian, this is the technique we will review today.
This seemingly superhuman feat takes advantage of the fact that JIRA Data Center runs on multiple nodes. Zero Downtime Upgrades allows you to put JIRA in a special “upgrade mode”, which allows nodes to be on different versions, then only bring down one JIRA node at a time while you perform the update. After the upgrade, you take it out of “upgrade mode” and let it start humming along on it’s new version!
Here is the doc we will be following that explains the what and how of doing a Zero Downtime Upgrade:
So, just because Data Center has fancy features, doesn’t absolve you from your own responsibilities. You will still need to go out and research what version makes sense for your instance, make sure it won’t impact any Apps, make sure your support systems are still supported under the new version, etc.
And I shouldn’t have to say this (I mean, I’ve pretty much said it the past two weeks now), but it is on you to thoroughly test each upgrade before putting it into production. JIRA is not a system that protects you from yourself. That is on you to do.
Also, “Zero Downtime” does not mean “Zero Risk”. It is still on you to make sure you have a good backup of the shared home, the database, and each node prior to upgrading the system. Remember our rule of thumb with upgrades:
“ALWAYS give yourself a path back to before you changed anything.”
Entering Upgrade Mode
So, you’ve selected your version, have everything ready to go, but where do you start?
To Enter the mythical “Upgrade Mode”, Head on over to Applications -> JIRA upgrades in the admin section (or hit “g” twice, then type JIRA Upgrades into the search bar that pops up).
That DB Collation Error will be taken care of shortly….
Click the “Put JIRA into upgrade mode” button, and you are off!
Upgrading each node
After putting JIRA into Upgrade mode, it’s time to do each node. Depending on the number of nodes you have, this can take a long time. Just remember, for this to be truly zero downtime, at least one JIRA Node must be up at a time.
I should also point out that every JIRA node that is still up will be getting the load of the nodes that are down for the upgrade. To me, if you have the capacity, it makes sense to build out extra nodes before a JIRA upgrade to make sure that no single node becomes overloaded while some part of them are down.
I won’t bore you with how to do the upgrade on each node – because I already covered it last week. At this point you treat each node like an individual JIRA server, download the installer onto each one, upgrade that system, then make any filesystem changes you need to.
If you have any customization in the JIRA install directory, you will need to copy these over as well. Remember, you can backup and restore the modified file – but during an upgrade that does carry a chance of breaking that node. It’s always a better idea to make a diff between the old and new files, identify the differences from your edits, and apply those edits to the new version of the document.
As each node comes back up, test it by bypassing the proxy and going to it’s IP directly. If you have customizations, you will get the warning about that, but then it should pop up with it’s new version.
Also check back at the proxy (System -> System info) to make sure each node is connected back to the cluster after upgrading.
Complete this for all remaining nodes, then continue on.
After you’ve upgraded all nodes
Once you’ve completed the install for all JIRA nodes, go back to Applications -> JIRA upgrades in the admin section. Here you will see a new button lit up, “Run upgrade tasks”. Go ahead and click it.
This will do all the edits necessary to the Database and Shared Home to get you running fully on your new version. As this is an important step, DO NOT SKIP IT!
Once complete, this will take you out of “Upgrade Mode” and return JIRA to normal operation.
Take this time to do your normal tour of the health checks, Apps, and basic functionality to make sure it is all working as expected on every node. Once you are satisfied, you’re done! All without any end-user interruption!
And if something went very wrong?
Well, depending on when it went wrong, you’ll change what you do.
If you have entered upgrade mode, but have not modified any nodes yet, you have the option to cancel out of upgrade mode. No harm, no foul.
If you’ve already started working on a node – but have not brought any nodes online under the new version, simply restore that node to the previous version, then cancel out and go back to test.
If – however – you do have nodes online under the new version, and want to back out fully to the old version, you are in for a bad day.
This is the point of no return for the upgrade, which means in order to return, you are going to have to bring the whole cluster down. Afterwards, you will need to restore the database, restore the shared home, and then restore each and every node that is running a new version. Then you can bring your cluster back online. Then write an email to all stakeholders explaining why you just brought JIRA down when they weren’t expecting it.
And this is why I keep telling you to test until you are confident. It won’t catch everything, but it will catch far more than not testing.
So, Listen – we have, like, a lot of nodes. Like, A LOT.
So, you are wondering if you can automate this on nodes somehow. Well….
You can automate a JIRA upgrade using tools like Ansible, Salt, etc. I actually have an Ansible job running of my Jenkins (sorry, no Bamboo here) which keeps my personal Confluence up to date without me having to upgrade it every time. However, I do not recommend you do this for production.
Atlassian is changing things all the time, and they don’t always get into the nitty gritty of what they are changing. That Confluence Upgrade job in Ansible I mentioned? It was actually broken for four months because Atlassian changed the URL you download from, and Ansible couldn’t make sense of the redirect, and I didn’t have time to debug it.
Humans are much better to adapting to these small changes. That’s why I prefer to always keep a human somewhere in the loop for a production upgrade on each node.
But, if that doesn’t dissuade you, I guess I can’t stop you. Just know I won’t be the one to tell you how to do it, just that it can be done.
And that’s Upgrading JIRA Data Center!
I hope you got something useful out of this guide. I think I’m done with upgrades for a while though – between work and the blog, I’ve upgraded 5 nodes across 4 instances in the past two weeks. So yeah – we’ll be covering something new next week.
Don’t forget to hop onto our discord community! There you can chat with fellow Atlassian Admins, get help, and see what’s new! https://discord.gg/mXuRsVu
And until next time, this is Rodney, asking “Have you updated your JIRA issues today?”
So, I was recently contacted by a good friend. He isn’t a JIRA Admin, at least by trade, but he’d kind of found himself in the role anyways. He’s facing an upcoming upgrade, and wanted to know if I wanted a side gig helping out.
Unfortunately, as I’m a consultant in my day job, there was some concerns about me doing the same thing on the side, so I had to decline. However, that doesn’t mean I can’t help out.
A JIRA upgrade is no trivial manner, but they happen everyday somewhere. If you know what you are doing, you can even automate it – to a point. I obviously don’t recommend automating a production upgrade, but for testing it wouldn’t be a terrible idea.
For this I’m going to use my JIRA Test instance. It’s running on CentOS and MySQL 5.7, and is currently running JIRA 8.4.3. For this I’ll be targeting the latest Enterprise release 8.5.3. Your system may vary a bit, but so long as you are on linux it should still be the similar enough that you can follow the post.
And yes, I am aware of you windows folks out there. I do owe you an install and upgrade guide too. But I only have one windows server license – and well…it’s in use. So that will have to wait for another day.
Data Center?
So, this guide here today is for JIRA Server. There are a few extra steps in a Data Center Upgrade, but the benefit to Data Center is you can upgrade without downtime, as you upgrade each node one at a time. So, I’ll cover a Data Center upgrade next week.
When you need an expert
So, my day-job is as an Atlassian Partner and Consultant. So – it’s my job to help people with their JIRA problems. And sometimes that includes helping people plan and execute their JIRA upgrades. So trust me, sometimes experts are needed.
So, when do you need an expert? Well…that depends. How confident are you in doing this yourself? Do you foresee problems you don’t know how to fix? What’s your support plan?
I’ve learned a lot by doing upgrades myself and getting help when I ran into a problem I couldn’t fix. But I had the benefit of working with Atlassian Premiere Support, and every time I came across a problem I’d work closely with them to learn what went wrong, what it’s supposed to do, and how to fix it.
That being said, I’ve been known to live off of adrenaline, and some people’s tolerance for this sort of thing is…less. Ultimately, there are no hard rules I can give you for when you need help. It comes down to your unique situation. But if you feel you do need help, don’t be afraid to reach out to an Altassian Partner. That’s what we are here for.
Planning it out
So, you want to go forward with the upgrade. What’s next? What are the questions you want to answer?
Well, the first question is “When should you upgrade?” This depends on your teams, but I find a good rule of thumb is to try to get an upgrade in every six months. This not always going to happen, but you shouldn’t let it go more than a year out. Updates to JIRA Server not only include new functionality, but a multitude of bug and security fixes as well.
However, you don’t always get your way. When I was working with Development teams, I’d always avoid an upgrade leading up to and immediately after a release. This is a time your teams depend on JIRA most, and having it down just isn’t an option, even for an hour.
Next question you have to answer is “What Version am I upgrading too?” To determine this, I’ll spend an hour or two with the JIRA Release notes. Not only will I go over each feature release, taking notes on which new features are available, but I’ll also look at each feature release’s upgrade notes for gotcha’s and changes. As I’ve stated in previous post, I give a heavy preference to an Enterprise Release.
After this, you should ask yourself “Will I need to upgrade anything else?” It doesn’t happen often, but sometimes Atlassian will deprecate support for a particular Database or JRE Version. So you should check the Supported Platforms Section to see if your systems are still good. Assuming you are, you can move on to testing. If not, take the time to get your supporting systems up to a supported version, then proceed.
And the last question you should be asking is “How will this affect my Addons/Apps?” Not every app is compatible with every version of JIRA. In order to account for changes in the JAVA API between different versions, a particular version of an app will only be compatible with a particular set of versions of JIRA.
Thankfully JIRA manages all this for you. You can even get a report of what apps will be compatible with the version of JIRA you are going to. Go to Manage Apps -> Manage Apps -> JIRA Upgrade Check
Here, you can select another release of JIRA, and it will give you a list of what apps are good to go as is, what apps will be good to go if updated, and what apps are not available for that version of JIRA. This will give you a heads up on what changes you need to make to be ready.
Now an App upgrade should be treated with the same respect as a JIRA Upgrade – even though it is far easier. App upgrades can change features your teams depend on, so always do them first in test and have your teams review that functionality to make sure it works as expected.
Test Test Test
So you have all the questions answered, what’s next? Well, Testing! Wait, is this déjà vu ?
Yep, I actually did cover how to go about testing last week. You should go back and read that. 😉
I’ll take a backup of my test instance, and upgrade it, then figure out what breaks. Then I’ll figure out how to fix that, document the fix, reset the instance, and test the full upgrade – including fix – again. I’ll rinse and repeat until I have a fully documented upgrade with no bugs.
This is always a bit of work, but I’d much rather break it in testing than in production. The last thing you want is to come into work the Monday after the upgrade to find a line at your desk. Trust me on that one.
Once you are happy with your test system upgrade, now comes the wait. You should give a week or two for end users to test the new version of JIRA before you put it into production. Not every user will test things out, but those that care the most will. So give them a chance to find what you might have missed.
Doing an Upgrade on System
So, this is the main event. Whether you are doing it as a test, or in production, the process should be the same. That being said, I suggest you practice it in test first to get a good feel for doing it yourself.
Now I shouldn’t have to say this after last week’s post, but here it goes. Backup now. Even if it is a test, give yourself an easy way back to a pre-install state. If you must do it manually, backup JIRA’s Home Directory and take a database dump. IF you can, just take a snapshot of the VM. But make sure you can undo this upgrade easily if you need to.
Either way, you will need to get the download first. Go to Atlassian’s JIRA Download Page. Once there, find the “All Versions Link”, then navigate to your chosen version.
Once you click the “Download” link, you will select “Linux 64 Bit” from the dropdown, click the check box to approve the EULA, then download the software. After this, you will need to transfer this file to your server via your favorite method.
Once on your system, you should have a file named “atlassian-jira-software-<version>-x64.bin”. We will make this file able to be run as a program using a chmod command. Be sure you are running the commands that follow as root.
chmod 755 ./atlassian-jira-software-*-x64.bin
Before we start the upgrade, we will need to shutdown JIRA.
systemctl stop jira
And once it’s stopped, we can initiate the upgrade
./atlassian-jira-software-<version>-x64.bin
The first option we face is just making sure we intended to start the installer. Take a second here and confirm the version matches what you are intending to upgrade to, then click “Enter”
After this, it will ask you the operation you intend to undergo. The Installer will detect that JIRA is installed on the system, and set the default option here to 3, which is upgrade. So press “Enter” to proceed.
The next option will ask you to confirm your installation directory. Take a second here and confirm this is correct. If it is, press “Enter” to proceed. If it is not, enter the correct directory, then proceed by pressing “Enter”
The next screen will ask you to backup your home directory. Technically, you should have already done this before you started (right?). Therefore, I often skip this step as it can eat up valuable time and disk space. And I already had a backup. But if you didn’t heed my earlier warning, this is your last chance to back up that information.
Next JIRA will check for modified files. These normally aren’t but two or three, with the most common suspect being <jira_install>/conf/server.xml. Open up a separate terminal window and copy these out of the install directory. Once you have these backed up, hit enter.
Next will give you a final checklist, and one last chance to back out. If you proceed from here, you are committing to changing the install directory, and will need to restore your backup to get back to a pre-test state. So hit “Enter” once to confirm you’ve completed the checklist, then hit “Enter” proudly once more. You are ready.
Here, JIRA will backup the current install directory, then replace it with a new one for the new version. For JIRA 8.5.3, it will not copy over any modifications you make, but we’ll get to that in a bit. Once done, it will ask if you if you want to start JIRA. Say “No”.
And JIRA is Upgraded….mostly. Time to deal with those modified files.
Re-adding modifications
So, you can just drop in the modified files you backed up during the install and have things right, correct?
Atlassian will on occasion make drastic changes to these files – changes that if not present, will cause JIRA not to run well – or at all. One such test I did on an upgrade, where I just copied over the files from a previous install, all I had from JIRA was a blank white page. Took me two days to figure out what I had did wrong.
Seriously, don’t be me. Or at the very least learn from my mistakes.
The easiest way we can tell how much has changed is with linux’s diff tool.
diff <backed up version of file> <current version from install directory>
If all you see are your modifications, then probably not a lot has changed. You should still use this as a road map to remake those modifications to the new files. Even a minor change can have an impact, no sense in risking it. Do this with all the files. Slow and steady keeps you out of trouble, after all.
It may also be worthwhile to store these modified files within a git repo – that way you can track what has changed with each upgrade across time. I’ve done it with various productions systems over the years, and it certainly cannot hurt.
After you’ve done this for all modified files, you are clear to start JIRA:
systemctl start jira
For the first start after a new install, I like to follow the logs to see if anything goes wrong.
tail -f <jira_home>/log/atlassian-jira.log
Assuming all is well, we can now log into JIRA’s UI to do the next step.
A note about modified files in JIRA 8.6 and above
As I noted when I looked at JIRA 8.6, the JIRA installer will start looking at your modified files and where it can, copy over the modifications to the new install. To be honest, I don’t like it.
If you’ve been in this business for any length of time, you too likely have a rather healthy skepticism to anything that happens “Automagically”. Even doing this, I’d be one to copy the modified files out during the install and compare the old ones to the new ones.
That being said, I admittedly haven’t done a JIRA 8.6 install yet, and their docs didn’t provide much detail on how the process works, so it could be fine. Just a warning to you in the future that an ounce of caution here might be worth a pound of cure later.
Updating Plugins
When you first get onto JIRA, it will warn you again about those modified files. Ignore the warnings to continue onto JIRA. From here, log in, then go to the Admin Console -> Manage Apps.
Come on Rodney – that isn’t good…
Given how App compatibility works, it’s likely you will have a few to upgrade. Take this time and upgrade these as well. Just click “Update” next to each one.
Be aware when you upgrade the “Atlassian Universal Plugin Manager”, you will need to refresh this page to proceed. That alone usually makes that plugin my first target.
Also be aware that any plugin update will require you to do a re-index. You don’t need to do a re-index for each update, but be sure you do one after you finish.
Wait…that’s not right…OH SHI
Now this guide assumes everything went well. As I’ve stated, everything doesn’t always go well. Hopefully this is in test and you have time to work it out. Even if it’s not, just take a second, and breathe, and relax. This isn’t ideal, but you got this.
IF this is in test, collect any logs files or support packets you can to your local system, then reset your instance back to before you started the upgrade. Try it again to confirm you did everything as you expected to. IF the problem reoccurs, you now have a pattern, and two data points.
Then open a ticket with Atlassian Support, and include everything you have, including a detailed account of the system you are running on, the version you were on, the version you are upgrading too, and all the files you got from the system.
If this is production, get what files you can, and execute your back out plan now. Restore the system to before the upgrade using the backups you have taken (don’t forget to also restore the Database) and bring it back up on the previous version of JIRA. There is no shame of backing out of a bad upgrade. There is shame on keeping your teams without JIRA “while you figure it out.”
Take those artifacts and open a ticket with Atlassian. Because this was production, and an issue that (presumably) didn’t pop up in Testing, it can often get a higher Business Impact than a test system problem.
And that’s it?
Yes, really, that’s it! You have JIRA Server running on a new version! Whether this is test or production, the above instructions should be sufficient to get you across the finish line.
At this point in my career, I’ve honestly lost track of how many Upgrades I’ve done. Using the method outlined above, I can usually complete an upgrade cycle – from kickoff to production upgrade complete, in about three weeks. Most of this time is to give end users a chance to test new features before they hit production.
So…humble brag. January is now the best month the blog has had to date, both in page views and visitors.
I cannot thank you all enough for coming by each week to see what I’m writing about. January has been an interesting month for me, with a lot going on. But being able to sit down at some point each week and share with you something about this tool has been both affirming and stabilizing. So thank you. Now lets see if we can make February even better!
Don’t forget that if you need help, we have a discord community. Stop by, ask questions, and chat with fellow Atlassian Admins: https://discord.gg/mXuRsVu
And until next time, this is Rodney, asking “Have you updated your JIRA Issues today?”
So, a bit of a backstory here. I was doing some experiments at work on running JIRA Data Center in Kubernetes using the official Atlassian containers when I noticed something odd. After loading the MySQL Connector and starting it all up, JIRA Setup kept telling me that the database wasn’t empty. I could see that it was, and per advice from a colleague, even double checked that the collations and char-sets were all correctly set.
Finally I isolated it down to the MySQL Connector. I had grabbed version 8.something, and Atlassian only supports version 5.1.48. And while this connector worked for JIRA 8.5.0, it apparently had some issues with JIRA 8.5.2 and 8.5.3.
This did get me thinking though. I went through the process of isolating the problem relatively quickly as I have had to do this fairly often in my career. But it isn’t the most intuitive thing to learn. So why not cover that this week!
Dev and Test
So, first thing: Friends don’t let Friends Test in Production. People are depending on that system being stable and there, and if you are mucking about in it constantly to “test” things, it will be anything but stable.
For all license tiers save the smallest, Atlassian also gives you an unlimited use Development License. And this is for both Apps and the main Applications. USE IT! If I.T. won’t give you another system, setup a VM on your desktop. IF they won’t let you use that, bring in an old PC from Home. There is no excuse for testing in production.
The most common setup I see is for a team to have two non-production instances of each platform: Test and Dev. Dev is your personal instance. This is where you can make changes to your hearts content, bring it up and down, upgrade it, reset it, whatever as much as you want. Break it? Won’t impact anything, and just refresh from Production. This is usually where I test “I wonder what will happen if I do this?” at.
Test, on the other hand, is your public non-production instance. You want to let a user test the functionality of a new App before purchasing it? Goes in Test. A user wants to add a new field? Put it in test and let them see what it looks like first. I usually like to refresh this from production on every JIRA Upgrade, but will do it sooner if we’ve made any big changes in production.
As a best practice, I also like to change the color scheme of JIRA for each instance, so you can identify which is which on site. My usual color scheme is to have the top bar be orange for Test, and Red for Dev. A few other things I do:
Separate out each instance to a separate DB Server
Make sure that if a given non-production server tries to talk to Production, it’s rerouted to the appropriate non-production instance instead. Often using /etc/hosts file.
DISABLE THE OUTGOING EMAIL SERVER
I definitely recommend you have both available. If you are only limited to one due to policy or budget, at least have a test instance. Your production instance will thank you.
But what about a non-production site for JIRA Cloud?
Okay – so I haven’t had to deal with this too often. BUT, you are also not the first person to ask, dear reader. Atlassian has a document actually outlining a few options you have to setting up non-production Atlassian cloud instances.
Take a snapshot and/or backup before changing anything
Before trying to figure out a problem or making a change, give yourself a way to get back to a pre-test state. If your instance (DB and all) is on a single VM, take a snapshot of the VM before starting. IF not, Take a tarball of your install and home directory, and while those are running take a database dump from your DB. Heck, if you can, take a file backup and a VM snapshot, do both!
Before I have your ESXI admins after me with torches and pitchforks, I should note here. The way I understand it, a snapshot setups up a way for ESXI to journal all the changes made to a system within a file, and revert back those changes. That means the longer a snapshot sits on a system, the larger it becomes. So always go back and remove a snapshot after you finish your testing. At the very least, it keeps things from getting messy.
This doesn’t only extend to a whole system. If you are changing a single file, make a copy of it first. That way you can go back to the file before you made any changes should the change prove catastrophic. The goal here is no matter what you are doing, always give yourself a path back to before you did it.
Isolate and make only one change at a time
This is probably the most challenging part of testing. For each run you do, you need to make only one change at a time. But what do I mean by change? Do I mean you should upgrade by changing one file at a time? Of course not!
The purpose of this is to isolate something enough to know what fixes or breaks it. So if you are doing a full upgrade, start by upgrading JIRA. Then check to see that it still runs as expected. Then make your changes to setenv.sh. Check again. Then server.xml. Then check again. Then upgrade the apps. Check again.
In the example I gave in the intro, here’s the changes I made each run when I found there was a problem with the DB:
Drop and Re-Setup the Database using a GUI Tool
Drop and Re-Setup the Database from command line.
Try a MySQL 5.7 DB instead of a MySQL 5.6 DB
Try JIRA 8.5.2 instead of JIRA 8.5.3
Try JIRA 8.5.2 with MySQL 5.6 instead of MySQL 5.7
Try JIRA 8.5.2, MySQL 5.6, with a different MySQL Connector – FIXED!
So you can see how each step I only changed one item. Yeah, it took me six runs to find a solution, but I now know it was for sure the MySQL Connector.
Yes, this adds significant overhead of bringing down and restarting JIRA each run. BUT – if and when something does break, you will know it was only the last thing you did that broke it. Likewise if something fixes it, you also know it was the last thing you did that actually fixed it.
Keep track of the changes you’ve made to each instance since the last Refresh
This is a bit of practical advice. Somewhere (Confluence), you need to have a document that shows in what ways each non-production instance has been changed since the last time you refreshed it from production.
Add a field? Add that to the doc. User tested an App? Document it. The idea is to have a journal to show what you’ve done, so that if you need to refresh it while a user is still testing something, you know where to find those changes to restore them.
And I get it – documentation is evil. Why spend time writing what you are doing when you can be doing more. This something I struggle with too! But this is a case where an ounce of prevention is worth a pound of cure.
Practice good Change Management on Production!
So, you’ve tested something in dev, put it before users in test, and now you are ready to put it on Production now. Enough delays, right?
Slow down there, friend! Production is sacred, you shouldn’t just run in there with every change.
Change control/change management is a complex subject – and honestly – hasn’t always been my strong suit. But it’s meant to keep you as an admin from your worst impulses. Annoying at times, I’ll grant you, but still a good thing overall.
The best way I found is to setup a board made of up of your Power Users, other Admins, and various other stakeholders as needed. Have them meet every so often (every other week seems to be the sweet spot here). If you have the budget for it, make it a lunch meeting and provide food. You are much more likely to get people to show up if they get to eat.
Then go over every change you want to make and gather feedback. They might spot a problem with a use case you hadn’t considered. But be sure to get a vote on each change before the meeting is over. Trust me, if you don’t structure and control the meeting, they will talk each point to death.
As a note here, there should be an exception to putting changes through the board during an emergency. If production is down, your first priority should be getting it back online as soon as possible. Then you can have time to retroactively put it through the board. For all non-emergency changes though, the change board is the valve to what you want to put into production.
Strictly this is not part of testing, but considering all, I didn’t want you to run off thinking testing was the last step. As with everything JIRA, it all works best when it’s a process.
And that is it!
You are ready to do some testing in JIRA. With the advice above, you are ready to maintain your JIRA Instances responsibly – or at the very least give yourself a way out of any sticky situations you find yourself in.
