How to test changes in JIRA

So, a bit of a backstory here. I was doing some experiments at work on running JIRA Data Center in Kubernetes using the official Atlassian containers when I noticed something odd. After loading the MySQL Connector and starting it all up, JIRA Setup kept telling me that the database wasn’t empty. I could see that it was, and per advice from a colleague, even double checked that the collations and char-sets were all correctly set.

Finally I isolated it down to the MySQL Connector. I had grabbed version 8.something, and Atlassian only supports version 5.1.48. And while this connector worked for JIRA 8.5.0, it apparently had some issues with JIRA 8.5.2 and 8.5.3.

This did get me thinking though. I went through the process of isolating the problem relatively quickly as I have had to do this fairly often in my career. But it isn’t the most intuitive thing to learn. So why not cover that this week!

Dev and Test

So, first thing: Friends don’t let Friends Test in Production. People are depending on that system being stable and there, and if you are mucking about in it constantly to “test” things, it will be anything but stable.

For all license tiers save the smallest, Atlassian also gives you an unlimited use Development License. And this is for both Apps and the main Applications. USE IT! If I.T. won’t give you another system, setup a VM on your desktop. IF they won’t let you use that, bring in an old PC from Home. There is no excuse for testing in production.

The most common setup I see is for a team to have two non-production instances of each platform: Test and Dev. Dev is your personal instance. This is where you can make changes to your hearts content, bring it up and down, upgrade it, reset it, whatever as much as you want. Break it? Won’t impact anything, and just refresh from Production. This is usually where I test “I wonder what will happen if I do this?” at.

Test, on the other hand, is your public non-production instance. You want to let a user test the functionality of a new App before purchasing it? Goes in Test. A user wants to add a new field? Put it in test and let them see what it looks like first. I usually like to refresh this from production on every JIRA Upgrade, but will do it sooner if we’ve made any big changes in production.

As a best practice, I also like to change the color scheme of JIRA for each instance, so you can identify which is which on site. My usual color scheme is to have the top bar be orange for Test, and Red for Dev. A few other things I do:

• Separate out each instance to a separate DB Server
• Make sure that if a given non-production server tries to talk to Production, it’s rerouted to the appropriate non-production instance instead. Often using /etc/hosts file.
• DISABLE THE OUTGOING EMAIL SERVER

I definitely recommend you have both available. If you are only limited to one due to policy or budget, at least have a test instance. Your production instance will thank you.

But what about a non-production site for JIRA Cloud?

Okay – so I haven’t had to deal with this too often. BUT, you are also not the first person to ask, dear reader. Atlassian has a document actually outlining a few options you have to setting up non-production Atlassian cloud instances.

• Atlassian Cloud: Staging and testing environment

Take a snapshot and/or backup before changing anything

Before trying to figure out a problem or making a change, give yourself a way to get back to a pre-test state. If your instance (DB and all) is on a single VM, take a snapshot of the VM before starting. IF not, Take a tarball of your install and home directory, and while those are running take a database dump from your DB. Heck, if you can, take a file backup and a VM snapshot, do both!

Before I have your ESXI admins after me with torches and pitchforks, I should note here. The way I understand it, a snapshot setups up a way for ESXI to journal all the changes made to a system within a file, and revert back those changes. That means the longer a snapshot sits on a system, the larger it becomes. So always go back and remove a snapshot after you finish your testing. At the very least, it keeps things from getting messy.

This doesn’t only extend to a whole system. If you are changing a single file, make a copy of it first. That way you can go back to the file before you made any changes should the change prove catastrophic. The goal here is no matter what you are doing, always give yourself a path back to before you did it.

Isolate and make only one change at a time

This is probably the most challenging part of testing. For each run you do, you need to make only one change at a time. But what do I mean by change? Do I mean you should upgrade by changing one file at a time? Of course not!

The purpose of this is to isolate something enough to know what fixes or breaks it. So if you are doing a full upgrade, start by upgrading JIRA. Then check to see that it still runs as expected. Then make your changes to setenv.sh. Check again. Then server.xml. Then check again. Then upgrade the apps. Check again.

In the example I gave in the intro, here’s the changes I made each run when I found there was a problem with the DB:

1. Drop and Re-Setup the Database using a GUI Tool
2. Drop and Re-Setup the Database from command line.
3. Try a MySQL 5.7 DB instead of a MySQL 5.6 DB
4. Try JIRA 8.5.2 instead of JIRA 8.5.3
5. Try JIRA 8.5.2 with MySQL 5.6 instead of MySQL 5.7
6. Try JIRA 8.5.2, MySQL 5.6, with a different MySQL Connector – FIXED!

So you can see how each step I only changed one item. Yeah, it took me six runs to find a solution, but I now know it was for sure the MySQL Connector.

Yes, this adds significant overhead of bringing down and restarting JIRA each run. BUT – if and when something does break, you will know it was only the last thing you did that broke it. Likewise if something fixes it, you also know it was the last thing you did that actually fixed it.

Keep track of the changes you’ve made to each instance since the last Refresh

This is a bit of practical advice. Somewhere (Confluence), you need to have a document that shows in what ways each non-production instance has been changed since the last time you refreshed it from production.

Add a field? Add that to the doc. User tested an App? Document it. The idea is to have a journal to show what you’ve done, so that if you need to refresh it while a user is still testing something, you know where to find those changes to restore them.

And I get it – documentation is evil. Why spend time writing what you are doing when you can be doing more. This something I struggle with too! But this is a case where an ounce of prevention is worth a pound of cure.

Practice good Change Management on Production!

So, you’ve tested something in dev, put it before users in test, and now you are ready to put it on Production now. Enough delays, right?

Slow down there, friend! Production is sacred, you shouldn’t just run in there with every change.

Change control/change management is a complex subject – and honestly – hasn’t always been my strong suit. But it’s meant to keep you as an admin from your worst impulses. Annoying at times, I’ll grant you, but still a good thing overall.

The best way I found is to setup a board made of up of your Power Users, other Admins, and various other stakeholders as needed. Have them meet every so often (every other week seems to be the sweet spot here). If you have the budget for it, make it a lunch meeting and provide food. You are much more likely to get people to show up if they get to eat.

Then go over every change you want to make and gather feedback. They might spot a problem with a use case you hadn’t considered. But be sure to get a vote on each change before the meeting is over. Trust me, if you don’t structure and control the meeting, they will talk each point to death.

As a note here, there should be an exception to putting changes through the board during an emergency. If production is down, your first priority should be getting it back online as soon as possible. Then you can have time to retroactively put it through the board. For all non-emergency changes though, the change board is the valve to what you want to put into production.

Strictly this is not part of testing, but considering all, I didn’t want you to run off thinking testing was the last step. As with everything JIRA, it all works best when it’s a process.

And that is it!

You are ready to do some testing in JIRA. With the advice above, you are ready to maintain your JIRA Instances responsibly – or at the very least give yourself a way out of any sticky situations you find yourself in.

Don’t forget to join us on Discord! https://discord.gg/mXuRsVu

Until next time, this is Rodney, asking “Have you updated your JIRA issues today?”