It is a glorious week, everyone! After nearly a year of running 8.5 as the last LTS version of Jira (formerly Enterprise Release version), last week Atlassian released version 8.13! This is likely the version that most serious customers will be running for the next year, so I thought I’d look at the new tricks and treats that Atlassian has in this release.
Wait, Long Term Support Release? What Happened to Enterprise Releases?
Yeah – it might be confusing if you don’t follow this stuff recently. Last June, Atlassian rebranded Enterprise Released into Long Term Support Released. The upshot is still the same – where possible, Atlassian will backport all bug fixes to this version for two years. That’s two years that you can make sure that your instance is safe and running without bugs – without having to worry about testing new features that may or may not break your company’s workflows. As a rule of thumb, I always go with the latest LTS release unless I have a very, VERY good reason to pick something else, like a critical feature that isn’t available yet.
So, What do we get for upgrading?
In a word, a lot. If you have been stuck on an LTS Version, things don’t hold still. There have been seven other versions of Jira released since the last LTS Release; each chalked full of new features. While I intend to go over a few of the ones I’m most excited about for Jira Software and Service Desk – there is no way that I can reasonably cover every new feature. So – I won’t. 😛
Instead, I’ll post the links here to the full changelogs for you to review yourself.
So, have you ever been searching for a user, and can’t tell which “John D.” is your John Steven Doe? I have – countless times. Jira finally fixed this in Jira 8.12 – giving us a User’s avatar, email, and Name in the search!
Image Courtesy Atlassian
Improved email notifications about mentions (From Jira 8.11)
So – I’ve touched on this before. Jira sends a lot of emails, and Atlassian has heard this cry. Now you can start seeing the benefits of this. Jira will no longer send you an email for every mention, instead adding it to a digest that you will receive. It will also move you up in the queue to receive the digest as soon as possible – but you won’t be flooded with mentions anymore.
Image Courtesy Atlassian
OAuth 2.0 Support for Incoming Mail (From Jira 8.10)
Both Google and Microsoft are planning to disable basic authentication. We don’t know when as they are coy about the timetables – but it’s coming. However, up until now, Jira only supported Basic Auth for email. Thankfully, They are ahead of the curve and putting in OAuth 2.o support.
Image Courtesy Atlassian
Dates on Future Sprints (From Jira 8.8)
Sometimes you need to plan ahead with your sprints. Up until now – you could only assign dates to the current sprint. That has changed. You don’t have to fill them out, but now have fields on future sprints for dates if so you want them.
Image Courtesy Atlassian
Anonymizing Users for GDPR Compliance (From Jira 8.7)
GDPR Audits as a Jira Admin is just painful. I don’t think I’m alone in thinking that. Now it’s just a little less painful. As an administrator, you can no completely anonymize a user’s record within Jira. This is a one-way operation, so if you do this by mistake, there is no recovering that user’s information, but it will honor a user’s “right to be forgotten,” which is a happy tick off the GDPR Audit.
Image Courtesy Atlassian
Jira Service Desk
Jira Service Desk support on Mobile (from JSD 4.12)
This feature is still in beta, but it’s exciting. Now you can have native Jira Service Desk support on the mobile Apps for iOS and Android. To get this rolling, you will need to enable the option in Jira, but once you do all your agents and users should be able to access the new features on their mobile devices.
Image Courtesy Atlassian
Multilingual customer portal and help center (From JSD 4.11)
Not everyone speaks the same language. If you run a portal used by a world-wide Enterprise, a person speaking any number of languages could be accessing your portal. Now Jira Service Desk comes with multilingual support in both the Portal and Help Center. This feature will allow your customers to access Jira Service Desk in the language they speak. You can even add custom languages or translations.
Image Courtesy Atlassian
Improvements to agent queues (From JSD 4.6)
This feature has been much needed. Atlassian has taken a look at the Agent queues and improved them. For one thing, you can now sort by columns to find the most critical issues to address first. And what’s more, the sort you apply is specific to your account, which means you won’t affect how your teammates sort their issues and find what works for you.
They’ve also added keyboard shortcuts to the queue screen so you can use arrow keys to hop between different issues in your queue.
Image Courtesy Atlassian
And that’s not all!
All told, if you are upgrading from the previous LTS version to this one, you will be getting some 27 new features for Jira Software and a whopping 42 in Jira Service Desk. Some of these are from Jira Core, so they are duplicated between the two. But that is a whole lot of new toys for your teams.
A new LTS version is something to celebrate. I mean, who doesn’t want new features on a platform you know you’ll be able to maintain for a bit? Back before Enterprise releases were a thing, I often felt you needed to upgrade every six months at a minimum to keep up. And then there came Premier support “blessed” versions to keep track of. Which weren’t a particular release, only a version that Atlassian’s Premier support team had tested and felt was a bit more stable than the rest. But honestly, this just felt like another thing to keep up with – you still needed to upgrade aggressively.
Having the ability to step back and not do a major upgrade as often is very much a load off of my feet. Yeah, there are still bug fixes to test and apply, but as they don’t include new features, the testing on those is much less demanding.
So what do you think? Are you planning to upgrade to the latest LTS Release? If so, what features are you looking forward to? If you enjoyed this post, help the algorithms by leaving a like and comment on social media of choice. You can find us on Twitter, Facebook, and LinkedIn, where we post new posts, news from the community, and anything I find interesting. You can also subscribe below to receive the latest posts directly to your inbox. But until next time, my name is Rodney, asking, “Have you updated your Jira issues today?”
So last week we took a dive into upgrading JIRA Server. So naturally this week it stands to reason that we will go over it’s big brother, JIRA Data Center. To be fair, doing an upgrade in Data Center isn’t much different than in Server. There are minor changes though that take full advantage of Data Center’s architecture, and those are what we’ll be focusing on today. We can sit here jabbering about it for a while, or we could get into it.
Zero Downtime Upgrades
So one of JIRA Data Center’s super-powers is the ability to stay online and completely functional while you are doing an upgrade! Called “Zero Downtime Upgrades” by Atlassian, this is the technique we will review today.
This seemingly superhuman feat takes advantage of the fact that JIRA Data Center runs on multiple nodes. Zero Downtime Upgrades allows you to put JIRA in a special “upgrade mode”, which allows nodes to be on different versions, then only bring down one JIRA node at a time while you perform the update. After the upgrade, you take it out of “upgrade mode” and let it start humming along on it’s new version!
Here is the doc we will be following that explains the what and how of doing a Zero Downtime Upgrade:
So, just because Data Center has fancy features, doesn’t absolve you from your own responsibilities. You will still need to go out and research what version makes sense for your instance, make sure it won’t impact any Apps, make sure your support systems are still supported under the new version, etc.
And I shouldn’t have to say this (I mean, I’ve pretty much said it the past two weeks now), but it is on you to thoroughly test each upgrade before putting it into production. JIRA is not a system that protects you from yourself. That is on you to do.
Also, “Zero Downtime” does not mean “Zero Risk”. It is still on you to make sure you have a good backup of the shared home, the database, and each node prior to upgrading the system. Remember our rule of thumb with upgrades:
“ALWAYS give yourself a path back to before you changed anything.”
Entering Upgrade Mode
So, you’ve selected your version, have everything ready to go, but where do you start?
To Enter the mythical “Upgrade Mode”, Head on over to Applications -> JIRA upgrades in the admin section (or hit “g” twice, then type JIRA Upgrades into the search bar that pops up).
That DB Collation Error will be taken care of shortly….
Click the “Put JIRA into upgrade mode” button, and you are off!
Upgrading each node
After putting JIRA into Upgrade mode, it’s time to do each node. Depending on the number of nodes you have, this can take a long time. Just remember, for this to be truly zero downtime, at least one JIRA Node must be up at a time.
I should also point out that every JIRA node that is still up will be getting the load of the nodes that are down for the upgrade. To me, if you have the capacity, it makes sense to build out extra nodes before a JIRA upgrade to make sure that no single node becomes overloaded while some part of them are down.
I won’t bore you with how to do the upgrade on each node – because I already covered it last week. At this point you treat each node like an individual JIRA server, download the installer onto each one, upgrade that system, then make any filesystem changes you need to.
If you have any customization in the JIRA install directory, you will need to copy these over as well. Remember, you can backup and restore the modified file – but during an upgrade that does carry a chance of breaking that node. It’s always a better idea to make a diff between the old and new files, identify the differences from your edits, and apply those edits to the new version of the document.
As each node comes back up, test it by bypassing the proxy and going to it’s IP directly. If you have customizations, you will get the warning about that, but then it should pop up with it’s new version.
Also check back at the proxy (System -> System info) to make sure each node is connected back to the cluster after upgrading.
Complete this for all remaining nodes, then continue on.
After you’ve upgraded all nodes
Once you’ve completed the install for all JIRA nodes, go back to Applications -> JIRA upgrades in the admin section. Here you will see a new button lit up, “Run upgrade tasks”. Go ahead and click it.
This will do all the edits necessary to the Database and Shared Home to get you running fully on your new version. As this is an important step, DO NOT SKIP IT!
Once complete, this will take you out of “Upgrade Mode” and return JIRA to normal operation.
Take this time to do your normal tour of the health checks, Apps, and basic functionality to make sure it is all working as expected on every node. Once you are satisfied, you’re done! All without any end-user interruption!
And if something went very wrong?
Well, depending on when it went wrong, you’ll change what you do.
If you have entered upgrade mode, but have not modified any nodes yet, you have the option to cancel out of upgrade mode. No harm, no foul.
If you’ve already started working on a node – but have not brought any nodes online under the new version, simply restore that node to the previous version, then cancel out and go back to test.
If – however – you do have nodes online under the new version, and want to back out fully to the old version, you are in for a bad day.
This is the point of no return for the upgrade, which means in order to return, you are going to have to bring the whole cluster down. Afterwards, you will need to restore the database, restore the shared home, and then restore each and every node that is running a new version. Then you can bring your cluster back online. Then write an email to all stakeholders explaining why you just brought JIRA down when they weren’t expecting it.
And this is why I keep telling you to test until you are confident. It won’t catch everything, but it will catch far more than not testing.
So, Listen – we have, like, a lot of nodes. Like, A LOT.
So, you are wondering if you can automate this on nodes somehow. Well….
You can automate a JIRA upgrade using tools like Ansible, Salt, etc. I actually have an Ansible job running of my Jenkins (sorry, no Bamboo here) which keeps my personal Confluence up to date without me having to upgrade it every time. However, I do not recommend you do this for production.
Atlassian is changing things all the time, and they don’t always get into the nitty gritty of what they are changing. That Confluence Upgrade job in Ansible I mentioned? It was actually broken for four months because Atlassian changed the URL you download from, and Ansible couldn’t make sense of the redirect, and I didn’t have time to debug it.
Humans are much better to adapting to these small changes. That’s why I prefer to always keep a human somewhere in the loop for a production upgrade on each node.
But, if that doesn’t dissuade you, I guess I can’t stop you. Just know I won’t be the one to tell you how to do it, just that it can be done.
And that’s Upgrading JIRA Data Center!
I hope you got something useful out of this guide. I think I’m done with upgrades for a while though – between work and the blog, I’ve upgraded 5 nodes across 4 instances in the past two weeks. So yeah – we’ll be covering something new next week.
Don’t forget to hop onto our discord community! There you can chat with fellow Atlassian Admins, get help, and see what’s new! https://discord.gg/mXuRsVu
And until next time, this is Rodney, asking “Have you updated your JIRA issues today?”
So, I was recently contacted by a good friend. He isn’t a JIRA Admin, at least by trade, but he’d kind of found himself in the role anyways. He’s facing an upcoming upgrade, and wanted to know if I wanted a side gig helping out.
Unfortunately, as I’m a consultant in my day job, there was some concerns about me doing the same thing on the side, so I had to decline. However, that doesn’t mean I can’t help out.
A JIRA upgrade is no trivial manner, but they happen everyday somewhere. If you know what you are doing, you can even automate it – to a point. I obviously don’t recommend automating a production upgrade, but for testing it wouldn’t be a terrible idea.
For this I’m going to use my JIRA Test instance. It’s running on CentOS and MySQL 5.7, and is currently running JIRA 8.4.3. For this I’ll be targeting the latest Enterprise release 8.5.3. Your system may vary a bit, but so long as you are on linux it should still be the similar enough that you can follow the post.
And yes, I am aware of you windows folks out there. I do owe you an install and upgrade guide too. But I only have one windows server license – and well…it’s in use. So that will have to wait for another day.
Data Center?
So, this guide here today is for JIRA Server. There are a few extra steps in a Data Center Upgrade, but the benefit to Data Center is you can upgrade without downtime, as you upgrade each node one at a time. So, I’ll cover a Data Center upgrade next week.
When you need an expert
So, my day-job is as an Atlassian Partner and Consultant. So – it’s my job to help people with their JIRA problems. And sometimes that includes helping people plan and execute their JIRA upgrades. So trust me, sometimes experts are needed.
So, when do you need an expert? Well…that depends. How confident are you in doing this yourself? Do you foresee problems you don’t know how to fix? What’s your support plan?
I’ve learned a lot by doing upgrades myself and getting help when I ran into a problem I couldn’t fix. But I had the benefit of working with Atlassian Premiere Support, and every time I came across a problem I’d work closely with them to learn what went wrong, what it’s supposed to do, and how to fix it.
That being said, I’ve been known to live off of adrenaline, and some people’s tolerance for this sort of thing is…less. Ultimately, there are no hard rules I can give you for when you need help. It comes down to your unique situation. But if you feel you do need help, don’t be afraid to reach out to an Altassian Partner. That’s what we are here for.
Planning it out
So, you want to go forward with the upgrade. What’s next? What are the questions you want to answer?
Well, the first question is “When should you upgrade?” This depends on your teams, but I find a good rule of thumb is to try to get an upgrade in every six months. This not always going to happen, but you shouldn’t let it go more than a year out. Updates to JIRA Server not only include new functionality, but a multitude of bug and security fixes as well.
However, you don’t always get your way. When I was working with Development teams, I’d always avoid an upgrade leading up to and immediately after a release. This is a time your teams depend on JIRA most, and having it down just isn’t an option, even for an hour.
Next question you have to answer is “What Version am I upgrading too?” To determine this, I’ll spend an hour or two with the JIRA Release notes. Not only will I go over each feature release, taking notes on which new features are available, but I’ll also look at each feature release’s upgrade notes for gotcha’s and changes. As I’ve stated in previous post, I give a heavy preference to an Enterprise Release.
After this, you should ask yourself “Will I need to upgrade anything else?” It doesn’t happen often, but sometimes Atlassian will deprecate support for a particular Database or JRE Version. So you should check the Supported Platforms Section to see if your systems are still good. Assuming you are, you can move on to testing. If not, take the time to get your supporting systems up to a supported version, then proceed.
And the last question you should be asking is “How will this affect my Addons/Apps?” Not every app is compatible with every version of JIRA. In order to account for changes in the JAVA API between different versions, a particular version of an app will only be compatible with a particular set of versions of JIRA.
Thankfully JIRA manages all this for you. You can even get a report of what apps will be compatible with the version of JIRA you are going to. Go to Manage Apps -> Manage Apps -> JIRA Upgrade Check
Here, you can select another release of JIRA, and it will give you a list of what apps are good to go as is, what apps will be good to go if updated, and what apps are not available for that version of JIRA. This will give you a heads up on what changes you need to make to be ready.
Now an App upgrade should be treated with the same respect as a JIRA Upgrade – even though it is far easier. App upgrades can change features your teams depend on, so always do them first in test and have your teams review that functionality to make sure it works as expected.
Yep, I actually did cover how to go about testing last week. You should go back and read that. 😉
I’ll take a backup of my test instance, and upgrade it, then figure out what breaks. Then I’ll figure out how to fix that, document the fix, reset the instance, and test the full upgrade – including fix – again. I’ll rinse and repeat until I have a fully documented upgrade with no bugs.
This is always a bit of work, but I’d much rather break it in testing than in production. The last thing you want is to come into work the Monday after the upgrade to find a line at your desk. Trust me on that one.
Once you are happy with your test system upgrade, now comes the wait. You should give a week or two for end users to test the new version of JIRA before you put it into production. Not every user will test things out, but those that care the most will. So give them a chance to find what you might have missed.
Doing an Upgrade on System
So, this is the main event. Whether you are doing it as a test, or in production, the process should be the same. That being said, I suggest you practice it in test first to get a good feel for doing it yourself.
Now I shouldn’t have to say this after last week’s post, but here it goes. Backup now. Even if it is a test, give yourself an easy way back to a pre-install state. If you must do it manually, backup JIRA’s Home Directory and take a database dump. IF you can, just take a snapshot of the VM. But make sure you can undo this upgrade easily if you need to.
Either way, you will need to get the download first. Go to Atlassian’s JIRA Download Page. Once there, find the “All Versions Link”, then navigate to your chosen version.
Once you click the “Download” link, you will select “Linux 64 Bit” from the dropdown, click the check box to approve the EULA, then download the software. After this, you will need to transfer this file to your server via your favorite method.
Once on your system, you should have a file named “atlassian-jira-software-<version>-x64.bin”. We will make this file able to be run as a program using a chmod command. Be sure you are running the commands that follow as root.
chmod 755 ./atlassian-jira-software-*-x64.bin
Before we start the upgrade, we will need to shutdown JIRA.
systemctl stop jira
And once it’s stopped, we can initiate the upgrade
./atlassian-jira-software-<version>-x64.bin
The first option we face is just making sure we intended to start the installer. Take a second here and confirm the version matches what you are intending to upgrade to, then click “Enter”
After this, it will ask you the operation you intend to undergo. The Installer will detect that JIRA is installed on the system, and set the default option here to 3, which is upgrade. So press “Enter” to proceed.
The next option will ask you to confirm your installation directory. Take a second here and confirm this is correct. If it is, press “Enter” to proceed. If it is not, enter the correct directory, then proceed by pressing “Enter”
The next screen will ask you to backup your home directory. Technically, you should have already done this before you started (right?). Therefore, I often skip this step as it can eat up valuable time and disk space. And I already had a backup. But if you didn’t heed my earlier warning, this is your last chance to back up that information.
Next JIRA will check for modified files. These normally aren’t but two or three, with the most common suspect being <jira_install>/conf/server.xml. Open up a separate terminal window and copy these out of the install directory. Once you have these backed up, hit enter.
Next will give you a final checklist, and one last chance to back out. If you proceed from here, you are committing to changing the install directory, and will need to restore your backup to get back to a pre-test state. So hit “Enter” once to confirm you’ve completed the checklist, then hit “Enter” proudly once more. You are ready.
Here, JIRA will backup the current install directory, then replace it with a new one for the new version. For JIRA 8.5.3, it will not copy over any modifications you make, but we’ll get to that in a bit. Once done, it will ask if you if you want to start JIRA. Say “No”.
And JIRA is Upgraded….mostly. Time to deal with those modified files.
Re-adding modifications
So, you can just drop in the modified files you backed up during the install and have things right, correct?
Atlassian will on occasion make drastic changes to these files – changes that if not present, will cause JIRA not to run well – or at all. One such test I did on an upgrade, where I just copied over the files from a previous install, all I had from JIRA was a blank white page. Took me two days to figure out what I had did wrong.
Seriously, don’t be me. Or at the very least learn from my mistakes.
The easiest way we can tell how much has changed is with linux’s diff tool.
diff <backed up version of file> <current version from install directory>
If all you see are your modifications, then probably not a lot has changed. You should still use this as a road map to remake those modifications to the new files. Even a minor change can have an impact, no sense in risking it. Do this with all the files. Slow and steady keeps you out of trouble, after all.
It may also be worthwhile to store these modified files within a git repo – that way you can track what has changed with each upgrade across time. I’ve done it with various productions systems over the years, and it certainly cannot hurt.
After you’ve done this for all modified files, you are clear to start JIRA:
systemctl start jira
For the first start after a new install, I like to follow the logs to see if anything goes wrong.
tail -f <jira_home>/log/atlassian-jira.log
Assuming all is well, we can now log into JIRA’s UI to do the next step.
A note about modified files in JIRA 8.6 and above
As I noted when I looked at JIRA 8.6, the JIRA installer will start looking at your modified files and where it can, copy over the modifications to the new install. To be honest, I don’t like it.
If you’ve been in this business for any length of time, you too likely have a rather healthy skepticism to anything that happens “Automagically”. Even doing this, I’d be one to copy the modified files out during the install and compare the old ones to the new ones.
That being said, I admittedly haven’t done a JIRA 8.6 install yet, and their docs didn’t provide much detail on how the process works, so it could be fine. Just a warning to you in the future that an ounce of caution here might be worth a pound of cure later.
Updating Plugins
When you first get onto JIRA, it will warn you again about those modified files. Ignore the warnings to continue onto JIRA. From here, log in, then go to the Admin Console -> Manage Apps.
Come on Rodney – that isn’t good…
Given how App compatibility works, it’s likely you will have a few to upgrade. Take this time and upgrade these as well. Just click “Update” next to each one.
Be aware when you upgrade the “Atlassian Universal Plugin Manager”, you will need to refresh this page to proceed. That alone usually makes that plugin my first target.
Also be aware that any plugin update will require you to do a re-index. You don’t need to do a re-index for each update, but be sure you do one after you finish.
Wait…that’s not right…OH SHI
Now this guide assumes everything went well. As I’ve stated, everything doesn’t always go well. Hopefully this is in test and you have time to work it out. Even if it’s not, just take a second, and breathe, and relax. This isn’t ideal, but you got this.
IF this is in test, collect any logs files or support packets you can to your local system, then reset your instance back to before you started the upgrade. Try it again to confirm you did everything as you expected to. IF the problem reoccurs, you now have a pattern, and two data points.
Then open a ticket with Atlassian Support, and include everything you have, including a detailed account of the system you are running on, the version you were on, the version you are upgrading too, and all the files you got from the system.
If this is production, get what files you can, and execute your back out plan now. Restore the system to before the upgrade using the backups you have taken (don’t forget to also restore the Database) and bring it back up on the previous version of JIRA. There is no shame of backing out of a bad upgrade. There is shame on keeping your teams without JIRA “while you figure it out.”
Take those artifacts and open a ticket with Atlassian. Because this was production, and an issue that (presumably) didn’t pop up in Testing, it can often get a higher Business Impact than a test system problem.
And that’s it?
Yes, really, that’s it! You have JIRA Server running on a new version! Whether this is test or production, the above instructions should be sufficient to get you across the finish line.
At this point in my career, I’ve honestly lost track of how many Upgrades I’ve done. Using the method outlined above, I can usually complete an upgrade cycle – from kickoff to production upgrade complete, in about three weeks. Most of this time is to give end users a chance to test new features before they hit production.
So…humble brag. January is now the best month the blog has had to date, both in page views and visitors.
I cannot thank you all enough for coming by each week to see what I’m writing about. January has been an interesting month for me, with a lot going on. But being able to sit down at some point each week and share with you something about this tool has been both affirming and stabilizing. So thank you. Now lets see if we can make February even better!
Don’t forget that if you need help, we have a discord community. Stop by, ask questions, and chat with fellow Atlassian Admins: https://discord.gg/mXuRsVu
And until next time, this is Rodney, asking “Have you updated your JIRA Issues today?”
