Creating a Custom Workflow

There are times as a JIRA Admin that you will get an odd request. Something that is well outside your strong-suit, but also something you can’t (or shouldn’t) say no to.

This happened to me at one point at my last job. the CFO of our small company came to me and my manager, saying that he had been told by our CEO that we needed to clean up our purchase order process, and that the CEO had told him to use JIRA to get it done.

Now, JIRA – strictly speaking – isn’t a finance tool. But, JIRA is a tool that can manage a process, and this was a process. So I told him what I’d need and what time frame we are looking at, and off we went to get it done.

I should note that I was working with a JIRA Software instance. There are some different ways to handle this in JIRA Service Desk, but I was limited to the version I was managing at the time.

Today’s post will be looking at the workflow that resulted from this, and how I came to the workflow.

Getting to know the pain points.

My Mom used to tell me a story about when she was in served in the U.S. Army. As a Sergeant, she developed a reputation as a “fix it” sort of leader. Put her in charge of a group, and she’d get them working together properly in no time.

One day, her Commander put her in charge of the company’s logistics group. Now my Mom specialized in Missile repair, not Logistics, and didn’t have a clue what they even really did. When she brought her concerns to her commander afterwards, he said this. “The team knows logistics, but you know how to lead. Trust them to know what they need to do and work with them to make sure it gets done.”

That story has stuck with me for some reason, and I find it oddly helpful in my role as a JIRA admin. You don’t need to know every little facet of how your teams do what they do, you just need to know what their pain points are and work to make sure they can get stuff done. If you become known as a problem fixer, your teams will let you know what problems need to be fixed.

Such as it was here. At University, finance was the one class that caused me the most problems. To me at least, it all seemed rather arbitrary, requiring lots of memorization. It’s the one class I had to work hard at to succeed in. But, as a JIRA Admin, I didn’t need to know every facet of Finance, just what were the problems they were trying to resolve.

So I sat down with the person the CFO loaned to me for this purpose. My first question was “What are the biggest problems in the current PO Process?”

I was told we had a paper-based process that required gathering actual signatures to move forward. However, as these papers were spread around multiple desks throughout the office, no one had any visibility into what had already started the process, what any one item was waiting on which signature, and double-ordering items because of this lack of visibility was a common occurrence.

Aha! There was the problem, and it’s one I knew I can fix with JIRA. With that, we move forward.

Drafting the workflow

With that discussed, I started asking about what the process should look like if it was to work as expected (ignoring the pain points mentioned earlier). I like to do this in front of a white-board – preferably with a few color options available.

As he goes through, I start building out a flow chart of what he is describing. It’s important to note that this won’t become the exact workflow, but it’s a way to work together to describe what the process currently is to come to a common understanding. So with these two bits of information, I go back to my desk and start working on the workflow.

What he described was a process where first the Manager would sign off on it, than the VP over that group. After VP approval was gotten, the Finance office would review it (usually the CFO himself). Here, depending on the price tag, the Finance office could approve it outright, or forward it on to the CEO for a final approval.

After all the needed approvals were gotten, it’d then be ordered, but no one was tracking when we received it, and lost items were also a thing that happened occasionally.

So with this, I came up with the following workflow:

It’s a bit spaghetti, but it works!

So, what’s going on here? Well, one of the biggest pain points was that no one had any visibility on who a particular PO was waiting on. As such, I made the approval chain status “Pending X Approval”. But rather than use actual names, I made them positions so I can re-use them (this will come up later).

I also simplified this by making the re-assignments automatic for most of the workflow. This is done using the “Update Issue Field” Post function, with the field being “Assignee” and the Assignee being selected directly.

As an added level of security, I also made it where the current assignee was the only person who could transition a status, using the “Only Assignee” condition:

This may lead you to the next logical conclusion: What’s stopping a user from assigning something to themselves and moving it forward anyways? Well…this actually happened. I had over-looked this in my first version of this workflow, and someone (who I won’t name) reassigned something pending CEO approval to themselves and moved it forward. Lets just say it’s NOT a good morning when you get to work with the CEO waiting at your desk.

There is a solution here, however. If you highlight a status in the workflow editor, you will see a “Properties” option. These status properties allow you to tweak how an issue behaves at a granular level. Much respect to the J-tricks blog for showing me what options are available and how to do this.

Here I would put the first person with the ability to reassign to the user that was auto-assigned the issue. The second person was myself. Being a small company, there wasn’t a tools team, so I didn’t need to worry about making this a group. If you do however, you’d use “jira.permission.assign.group” as the Key and your group name as the Value. Be aware that this puts a magnifying glass on you, so people will know if you abuse the system yourself. Also, if you have more than one user or group keys, you will need to numerate them like in the example above. Otherwise JIRA will get confused. Odd, but true.

This works well for one group, but as you may know, JIRA doesn’t really know or care who a particular person’s manager is, or who their VP is. This honestly caused me the most problems.

However, the solution I came to was to have multiple workflows, one per group, and assign them to different custom issue types name for that Group. So we initially had “IT Purchase Orders”, “Facilities Purchase Orders”, “Engineering Purchase Orders” Issue Types. This allowed each workflow to be tailored to each specific group and track through their management chain accordingly. This is why using generic Statuses was so important earlier, as it allowed the same statuses to be used by all of these workflows.

After the approval chain, I added a couple of more statuses describing the wait for it to be ordered and delivered. This was to track an issue all the way to make sure it was actually delivered as expected – as this was a stated problem as well.

One last note I’ll make on this workflow. I added an open step at the beginning to give the requestor one last chance to review everything before sending it into review. This also gave me a status to send things back to if an approver didn’t want to reject a request, but felt it still needed more information from the requestor. However, this caused some requests – especially when I first put this process in place – to sit in this opening status because it was never actually sent for approval. You can – as an option – send things from issue creation directly to Pending manager approval, then just have an offshoot status for requesting more information that goes back into Pending Manger Review. I personally don’t like this option, as I like the “Are you sure?” approach to submitting requests. But, on the whole sending items directly for approval is technically sound and a valid approach.

Closing thoughts

Now, this wasn’t the whole project. I still had to go through three more meetings about what fields were needed, but that is another blog post altogether…probably something along the lines of “What to do when you and your users disagree.” But before I go, I wanted to thank my previous company and my former Manager. I actually had to get her approval to be able to share this story with you, as well as her encouragement that lead me to originally start this blog. So, until next time, this is Rodney, asking “Have you updated your JIRA Issues?”

Posted in Uncategorized

Leaving the Breadcrumbs: how to adjust Logging.

So, we’ve discussed how to read your logs, and what impact changing them will have on your disk drives. So how do we go about changing logging levels and tuning logging? That is what we are going to discuss today. So without much preamble, lets get into this.

Should I adjust my logging levels?

If I’m being honest, No. Well, that was a quick blog post, I’ll see you next week!

Actually, let me explain. The default levels are sufficient for 99% of admins out there. It’s a good balance of what you need to know to diagnose issues without filling up your disks. Typically, I only recommend people only adjust these levels if asked to do so by Atlassian or App Vendor support.

However, it’s still important that you know how to do so when asked. And with a bit of homework you might even be able to adjust them and find your answers before you have to get support involved. My advice though is to do so carefully if you choose to.

I just need to note something here, and it’s something I forgot to mention last week. Debug level logs can sometimes capture passwords in the log file. I should not have to tell you why that could be bad. However, this is just one more reason why you should really think twice about capturing any logs on the Debug level.

Temporary vs Permanent

So, the first question you need to answer is do you need this change to be permanent or temporary. Atlassian does give us two ways to change logging – one via the Admin console in the web UI, which will only last until your next JIRA restart. As such I label this the “temporary” option. The second option is by changing some files within the JIRA install directory, which will persist across application restarts, and as such I label a “permanent” change.

As I tend to recommend you stick to defaults, so without any deeper context, temporary is my go to answer. However, as with all things Atlassian, context is everything. Lets say you’ve had some problems with a new app you’ve installed onto your instance, and it’s causing the Application to restart regularly. This is a time you’d want to look at a more permanent solution, as you’re not going to capture those detailed logs while JIRA loads up otherwise.

If you are working with Support, they will almost always tell you to go the temporary route. However, no matter what I say, my advice is to follow their advice. Seriously, they are scary good at what they do, and they are not going to steer you wrong.

Making a Temporary change to logging.

To change something temporarily within the logs, we’ll need to go to the Logging and Profiling section of the JIRA Administration Console. Once there, you’ll find it under System -> Logging and Profiling.

Note: You will need System Administrator global permissions to be able to see this section.

As a pro-tip, you can also find any admin page from anywhere in JIRA by hitting the period key on your keyboard, then typing the page you are interested in. For this to work, your cursor must be outside a text input box of any kind.

Once here, you’ll see different sections:

  • Mark Logs
  • HTTP Access Logging
  • SQL Logging
  • Profiling
  • Mail
  • Default Loggers

Today we’re going to be primarily interested in the Mark Logs section and the Default Loggers section. Everything else is available to turn on, but remember that these logs will only run until the next time you restart JIRA.

Mark Logs

The first section here is where you can add a comment into your logs. You can also roll over your logs. A roll over is where JIRA will increment the end number on all existing rolled over logs, copy the current log file to atlassian-jira.log.1, then start a new atlassian-jira.log file.

Both of these techniques are great for trying to mark a section of logs before doing some operation or test. In fact, I made great use of this functionality to do my testing last week on log sizes. I can search a ten minute manual search for where you started doing something to a “It’s right here” instant search.

Default Loggers

The next section of interest is the Default Loggers section. This has the familiar logging levels shown, on a per-package bases. A package is an individual class or object within the JAVA code, so each of these will adjust the logging on a specific aspect for JIRA.

This unfortunately is an exhaustive list, and I don’t have time to write up what each of them do (assuming I even know that!). However, most of these come down to logic, and with a bit of searching you can find something.

That is to say I won’t offer any guidance. If I’m being honest, it’s amazing how many times JIRA problems turn out to be App problems. So I’m going to quickly discuss how to add and adjust logs for an App here.

Our first step is to go to our App Manager, then expand the information for the App we are interested in.

Really should update that…

Look for the App Key, highlighted above, and copy it. Then head over to Logging and Profiling, and under Default Loggers, click “Configure logging level for another package”. Past the App key into the Package name section, then select your logging level. Click Add and boom, you are now logging for that particular App.

Permanent Logging Changes

To change logging levels, you will need to find the <jira-install-dir>/atlassian-jira/WEB-INF/classes/log4j.properties.

Here we can change a number of things, including log levels of various packages, where the logs are, and so on.

Super Important Note: Before making any changes to this file. Make a copy and save it in a safe place. Your future self will thank you.

To change the logging level, you first need to know the package you are adjusting for. You will either have gotten this from Support or the App Manager section of JIRA. Then we’ll look for log4j.logger.<package-name>. You should see something like this:

log4j.logger.com.atlassian = WARN, console, filelog
log4j.additivity.com.atlassian = false

To adjust the log level, change “WARN” to any of the other logging levels, then save. After you restart JIRA you should see the logging level change reflected. This goes with any change to this log file – you will need to restart JIRA to see any changes.

And that’s logging, done.

So I’ve actually had a lot of fun going back over this subject matter. It was also the first reader-requested topic, so that is amazing in and off itself. So what do you guys think I should cover next? Leave a comment here or on LinkedIn and if it’s good, I’ll cover it. So until next time, this is Rodney, asking “Have you updated your JIRA Issues today?”

Pile of Breadcrumbs, how logging levels impact JIRA’s logs

Well, seems we are back on track. Last time we looked at logging, we went over how to decode the information that was in the logs. During that piece, I made the following claim:

If you set everything to Debug before you leave on Friday, by the time you are sitting down for dinner on Saturday, you’re going to be paged for a full disk.

Following the Breadcrumbs: Decoding JIRA’s Logs, 30 Oct, 2019

Well, this got me to thinking….exactly how much does JIRA’s logging level impact log size. To figure this out, we need a bit of SCIENCE!

Ze Experiment!

So here’s what I’m thinking. We take a normal JIRA instance, and we do a set number of tasks in there, roll over the logs, then see what size they are. Rinse and Repeat per log level.

So the list of tasks I have for each iteration is:

  • Roll Over the Logs
  • Logout
  • Log in
  • Create Issue
  • Comment on Issue
  • Close Issue
  • Search for closed issue
  • Log into the Admin Console
  • Run Directory Sync
  • Roll Logs

Between the last Roll over and the first one of the next iteration is when I’ll capture log size and adjust the logging levels.

Now to have as much of an apples to apples comparison, I’ll need to limit the background tasks as much as possible. The biggest one I can think of is the automated directory sync, which will need to be disabled for the test. However, as this is a regular activity within JIRA, I’ll be including a manual directory sync to capture that into the data set.

I’ll also need a control to measure what JIRA does by default, but that will be my first run. To make sure I can return to defaults later, I’ll be adjusting the logging levels in Administration -> System -> Logging and Profiling. Changes to the logging level here do not persist over a restart, so this should be ideal. So, without further adieu, See you on the other side.

The Results

Well, That was an adventure. I ended up taking a backup of the log4j.properties file and changing it. Turns out changing > 110 settings by hand one at a time is not very time-efficient. Changing the log4j.properties file dropped the number that I had to change manually to ~30.

I tried to be as consistent as possible with each run. That means I had the same entries for each field on the issue I created, same comment, same click path, etc. I even goofed up my search on the control run (typed issueky instead of issuekey), and I repeated that mistake for each run afterwards.

Another note I want to make is that there was one setting I could not change. Turns out that if the com.atlassian.jira.util.log.LogMarker object is anything but Info, JIRA will crash when you go to roll over the logs. Oops!

Conclusions

I think even considering all that, my assertion still stands. This was one person doing somewhat normal JIRA tasks. With that the Debug was still almost 1000x the Info log. Fun fact, it rolled over the logs automatically four times. Now multiply that by how many people are using your instance, and how many times they will be doing these kinds of operations, over and over again. Even if they aren’t, there are automated processes like the Directory syncs that will take up log space. It will definitely add up fast.

However, I think the bigger consideration here is never take anything at face value. Yeah, I’m an expert, but even I was just parroting something I had been told about logging. Now I have the experiment and data to backup my assertion. Don’t be afraid to put things to a test. You might discover where some advice isn’t right for your environment. Or you might find out why things were said, and become that much more knowledgeable . The point is to always be learning.

So until next week, this is Rodney, asking “Have you updated your JIRA issues today?”

JIRA Service Desk Vulnerability – 06 Nov 2019

Yes, I know what I promised you guys last week. And more information about logging is coming, but sometimes there are things that are more important. This is one of them.

I’ve spoken briefly about security advisories before. Well….Atlassian has just announced a Security Advisory for JIRA Service Desk, that affects both Server and Data Center versions. As such, I figured that was worth breaking into our regularly scheduled programming to discuss it in more detail.

So, what’s going on already?

Atlassian has released several patches that solve a significant vulnerability in JIRA Service Desk that can let people bypass authorization. For any service, this is not a great bug to have. I mean, they aren’t leaking credentials in clear text, but this is almost as bad.

HOWEVER, there are workarounds to mitigate this bug, as well as fixes released to resolve it entirely. I’ll be going through it in more detail, but to hear it straight from the horse’s mouth, here’s the link:

How do I know if my version of JIRA is affected?

First off, this bug only affects JIRA Service Desk. Which means if you are running JIRA Software or JIRA Core, You’re good. Also, if you are using Atlassian’s cloud offering, you are also good. This bug only impacts servers running JIRA Service Desk under their Server or Data Center offerings.

Furthermore, it doesn’t impact every version. Below I’ve listed out the Versions that are Affected:

Versions with Bug Present

  • Any version before 3.9.17
  • All of 3.10.x through 3.15.x
  • 3.16.x before 3.16.10
  • All of 4.0.x through 4.1.x
  • 4.2.x before 4.2.6
  • 4.3.x before 4.3.5
  • 4.4.x before 4.4.3
  • 4.5.0

If you are running any of the above versions, below are the versions of Service Desk where this is fixed. As always, I recommend you go with an Enterprise Release, which the latest one for Service Desk is Version 4.5.1

Versions where the Bug is fixed

  • 3.16.10
  • 4.2.6
  • 4.3.5
  • 4.4.3
  • 4.5.1

So…exactly how worried should I be?

Well, that depends. Is you system exposed to the internet? You probably should be putting mitigations steps into place right now rather than reading this, honestly. No I mean it, close this window and go fix it now! I’ll be here when you return.

Even if you are not, It’s always shockingly easy to gain access to places people think are “secure”. That is to say you can’t always assume that just because you have a good firewall between your system and the internet, that someone can’t bypass that by just walking into your office and finding an unlocked computer. Looking at you Darren!

A good thing about this bug is that it has a work-around that you can put in place if you can’t upgrade right away. All you have to do is add the following bit to the file <jira-install-dir>/atlassian-jira/WEB-INF/urlrewrite.xml

<rule>
    <from>/servicedesk/.*\.jsp.*</from>
    <to type="temporary-redirect">/</to>
</rule>

After you add the above rule to the file, be sure to restart JIRA so that the changes are picked up, and you are safe until your next upgrade…or vulnerability disclosure, whichever happens first.

Assuming you are not willing to mess with the Atlassian Install directory (I cannot blame you), you can also make some changes to your proxy or load balancer to mitigate the bug. However, this is only as secure as your users’ inability to bypass the proxy, so your mileage may vary.

Seriously though, you should upgrade sooner than later. Just saying.

Why should I care?

Unfortunately for us, black hat hackers and other ne’er-do-wells have figured out that enterprises love running Atlassian Applications. Which means they are now actively looking for vulnerabilities to use against these applications. Why develop a niche tool that will only work against a handful of targets when you can develop one that will work on just about every target?

However, Atlassian’s Bug Bounty program means that the good guys are just as motivated to find these bugs and report them responsibly to Atlassian before the bad guys can find them.

But, that doesn’t mean you can slack off. People on both sides of ethical line are finding bugs, and even when a bug is found by the white hats, black hat hackers are still weaponizing them. If you’re lucky they are only installing a cryptocurrency miner on your system. You don’t want to be called in because the entire company was ransomware’d, with your system being patient zero.

Also, can ransomware be a verb? Oh well, it is now!

Well, that’s all for this week

I am actually delaying this post a bit so that I don’t disclose this before Atlassian does (Responsibility!), but I also want to bring your attention to something.

Atlassian opened up registration for Summit 2020. If you haven’t been to one before, I have one bit of advice. Do. It.

No, I’m serious! Beg your manager and their manager to have your company pay for it. If they won’t, take a vacation and pay for it yourself. It is ABSOLUTELY worth it. Between the Keynotes and talks, you will learn not only what the best practices are, but what’s coming down the pipeline. Last time I went, they announced native iOS and Android apps for the cloud versions. That was some exciting news to return to the office with.

You will also network with so many other people in the trenches just like you. Trust me, when you’re doing the day to day as the lone Atlassian Admin, it’s things like this that lets you know there are people who get it.

And don’t even get me started on Summit’s Bash. You’ll want to go 😉

For more details, here’s the link to the website:

Who knows, maybe you will even bump into me.

However, if you can’t go, don’t fret. Altassian has always been good about posting not only their keynotes, but all talks and presentations onto YouTube. I pretty much block off that week to catch up on everything when I can’t go, so you won’t miss anything critical.

So, until next week when we return to our look at Logging in JIRA, this is Rodney, asking “Have you updated your JIRA issues today?”