Jira Blunders!

Last week I got a call from my family. Trust me; this does have something to do with Jira. My Mom told me that my younger brother of 17 was having problems with his Chemistry homework, and she wanted me to go over it with him. Being 17, he protested saying he didn’t need my help because he “had used a calculator.”

When I put my foot down and brought him into a Zoom meeting, I found that he had used an incorrect formula. This situation meant that while his math work was right, the answer was still wrong. I sat him down and explained his error to him, to which he stated he had hurriedly copied down the formula wrong at the end of a class.

I then asked him, “What did you learn?” He replied he needed to double-check his formulas. So what does this have to do with Jira?

We all make mistakes. It’s both very human and part of the learning process. However, not taking away a lesson on it is just a waste, which I was trying to impress on my little brother. The same goes for Jira. There will be blunders. This week, I am reviewing some blunders both I and colleagues have made to get a chuckle and help teach some important lessons.

Always Double-check system commands

So this didn’t happen to Jira – but it just as easily could have. I was working on a Perforce upgrade on a system. As part of this upgrade, I needed to upload a new directory to the server, then change its permissions. This server was a Linux-based system, so I was working on the command line as per my habit. 

So I uploaded the directory and its files, and then I went to change the permissions so that the perforce user could access those files. This is where disaster struck.  

This was the command I meant to run:

chown -R perforce:perforce ./

And this was the command I ran:

chown -R perforce:perforce /

Catch that change? One tiny little dot missing. For those who may not know, a “./” in Linux is shorthand for, “The folder I am currently in.” However, “/” , when paired with a -R, is shorthand for “Every file on the system.” This slight error meant that every single file on the system now belonged to the perforce user. But it gets worse. So, so much worse.

This exercise took place during work hours and was meant to stage some files ahead of a weekend upgrade on Production. Which meant I had just made a catastrophic error, on Production, in the middle of the workday.

I eventually realized my mistake when the change command took much longer to process than it should have. Even then, I’m embarrassed to say it had processed a fair bit of the filesystem before I realized my mistake. So, damage control mode. I changed most of the filesystem back to root ownership, save for the processes and files I knew to be managed by other users. Somehow I had it all back to normal relatively quickly. The miracle is that no one attempted a push to Perforce during all this, so my mistake went otherwise unnoticed.

So, what lesson did I take away from this? Well, if I am on Production, I do not free-hand commands. I write them out ahead of time, then copy & paste them into the terminal while working. Even if it’s a minor task, Production is sacred ground – treat it as such.

Label Your instances Carefully

This one was another mistake of mine that took place in the middle of the day. We were having some problems with our Jira system, so I was engaging with Atlassian Support to see if we could resolve them. Atlassian had helped us identify a possible culprit, but we needed to test changes in a Development environment before pushing changes to Production per our policies.  

However, our change controls for Development was – how do I put this – non-existent. Which meant I was free to go into Dev and make these changes whenever. So I decided the 11:30 AM on a Tuesday was a perfect time!

Now, when I’m a Windows PC, and I need to connect to a Linux Server, I’ll use mRemoteNG. It’s just a handy tool to manage all your system connections and work with individual systems as needed. However, this is how I had labeled and set up my connections at the time.

Can you see where this is going? Yep. I connected to “Dev” and started making my changes. I double-checked all the changes against Atlassian’s ticket, and once I was happy, I restarted Jira. Then I started getting questions.

“Hey, is Jira down?” “What happened to Jira?” “Is Jira down for you too?”

With a cold sweat, I looked over to my connection and realized my mistake. I had just made these changes to – and restarted – Production. And given our hardware at the time, Jira would be down for another two or so minutes.  

If there is something I do well, it’s handling downtime. I let my team know what had happened, and that Jira was coming back up. Yes, embarrassing, but I would have time to be embarrassed later. They ran interference for me while I did the last few steps to bring Jira back up and running. Crisis averted.  

So what did I learn? That day I learned that I needed to do a better job of labeling Production and Development systems. I also needed to put safeguards to ensure that I realize that I’m connecting to Production. After that day, this is how I started organizing these systems. 

I also removed the stored password from the Production systems so that I would have to enter it every time I connected to these servers. This process would serve as an extra layer of “Something is not right here,” which would hopefully prevent me from making that mistake again. And well, so far, it’s worked!

Oh, and if you were wondering, the new settings did fix our Jira problems, so there’s that happy ending.

Leaving your email on while Load Testing

This one came from a colleague at work. I’ll often ask for input or bounce ideas off my fellow East Coast consultants at Coyote Creek. This entire article started as one of their ideas!  

Anyways, one of my friends was doing some load testing on Jira for a client. He had just cloned their Production system to create a Development environment; that way, he could do his testing without impacting users. All per best practices, so nothing wrong there.

So he then turned on JMeter and started creating hundreds of issues on Jira. However, when he cloned Production, he had forgotten to turn off outgoing email, which meant that now users were getting flooded with emails. When they’d go to find these issues in Jira, they’d not exist. Word eventually filters back to him, and he shuts down the test.

So, what could we learn from this blunder? That’s simple. It is considered best practice to turn off email on any Development system you have. Don’t get me wrong; there will be times you need to test things related to email. When this happens, I turn email back on only long enough to run my test, then turn it immediately back off. It’s just not worth the user’s confusion.

And that’s it for this week!

What blunders have you made with your Jira instance? Have you any lessons learned you’d want to share? Then be sure to leave a comment on this post, and I might include it in a follow-up! And if you enjoyed this post, subscribe below to get new articles delivered right to your inbox!  You can also follow the blog on Facebook, LinkedIn, and Twitter to get the latest from us.

There are a few things I want to call your attention to this week. I want to start featuring interesting posts, webinars, or webpages I run across – and I have a few to share this week!

Let’s start with this blog post from Notes From Kris, which explains the various resources you have access to from Atlassian. Having all of these in one place is handy – especially if you are new to the Atlassian Ecosystem.

The next is a fun JQL quiz from Elements, which is an Atlassian Marketplace partner. I haven’t used any of their apps before, but they are definitely on my radar now. The JQL quiz was rather fun, but also very challenging. Give it a try and let me know what score you get!

And lastly, a webinar from Atlassian on “How to become an Atlassian Community Leader.” On top of my current goal to become an Atlassian Certified Master, I am also working towards a Community Leader status, so I plan on attending this webinar.

I hope you found something useful in this section. If you have, let me know, and I’ll keep including exciting things I see here! But until next time, my name is Rodney asking, “Have you updated your Jira issues today?”