Its all about them schemes

Last week we took a look at cleaning up your fields to help improve your system performance. So now we move on to our next Atlassian TAM suggested topic: How to standardize workflows and screens. But before we dive in…

Last week was amazing. I had a lot of engagement on this topic, and even had two more people come forward with requests for more topics to write about. I’m always up to do a reader suggested topic, so if you have something you want to see covered, let me know! It may be a few weeks before I can write on your topic (For once, it’s nice to have a backlog!), but I’ll definitely get to it. You can use the contact form on the blog’s website, or email me directly at contact@thejiraguy.com.

Anyways, lets get into it.

So, let me ask: Why would you want to standardize?

Fair question. This is going to be a lot of work, negotiation, and may come down to you telling them this is how it’s going to be. Why bother?

What’s the benefit that will make all this worthwhile? Well, the first benefit it helps teams within your organization work better together. If a person has to move from Team A to Team 1, they don’t have to relearn how to use JIRA. They’ll see the same fields and the same workflows. They’ll know how to use the tool as they’ve always have and be able to focus on getting to work that much quicker.

Secondly, It will help with reporting. With all the teams using the same workflows and fields, we have a consistent set of data across all teams with which to run reports and measure performance. While you should NEVER compare teams’ velocities to each other (as each team will value story points differently, therefore you will never get an oranges-to-oranges comparison), you can measure most other things consistently across the teams if they are all working from the same field set.

And the third, and I’d argue the most important reason to consider standardizing your schemes is your sanity. Look, there is only so much JIRA Admin to go around. If you are lucky, your organization has two or three of them to help out – but that’s usually for the largest organizations. Most small and medium businesses, it’s on you. Having standardize schemes stops you from having to play “What impact will this change have?”. You’ll know, and be able to get on with it that much faster.

But what about the exceptions?

Look. There will always be exceptions. A process or team that doesn’t adhere to normal business models. My Approvals workflow is such an example. But it’s your job to make it just that: An Exception. If you standardize your workflow, any change that is proposed by a team using the standard schemes should be measured on “How will this benefit the entire group?”. If it’s a benefit, it should be applied to the entire scheme. No breaking off a scheme to apply to a specific team. It’s all or none.

This, in and off itself, can help limit JIRA bloat. If a team has to defend why their change will benefit everyone, not just themselves, they will sometimes decide it’s not worth it.

So, where do I get started?

Honestly, I’d start with the screen schemes. And the first step is to talk with each and every one of your teams. Figure out what fields each one is using, and how they are using them. This will be time consuming – just a fair warning. But to do this successfully requires you to put in the work and really get to know your teams and how they work within JIRA.

Once you have that, sit down and compare your notes from all the teams. (You did take notes, right?) Find similarities, and try to get down to the fields you can get away with to allow as many teams to work as they already do as possible. Try to avoid fields that are used by only one team – opting instead to put that information in another field if possible, or the description. Take this time to use your test instance to make the changes. This will be both your testing and give you a way to demonstrate the change in a real way in our next step.

Now that you have your proposed standardized fields, this next step is probably the most important. Go back to each team, and present the new scheme proposal to them.

No, Seriously. Review what changes will impact their team specifically, and explain why you are making each change. Get feedback. Listen. Negotiate. Being willing to work with each team on this process is how you build buy-in, and it’s how you will make this process a success. Not everyone will be a 100% happy, and you may have to add some fields back in, but you can get them to a point where they can at least agree to the change. Again, I know this will take time, but having gone through this process myself, it is absolutely required.

So, you have con-census and buy in from your teams, and you have the changes ready to go in your test instance. You are now ready to push this to production..which in vanilla JIRA isn’t going to be fun. Without any plugins, you will need to manually recreate the new schemes in your production JIRA Instance, then assign all the appropriate projects to it. Or – you can use a tool like Botron’s Configuration Manager to copy over the new config as it exists from your test to production. Either way, I usually schedule a full change window for this during a relative down period for the instance (which invariably ends up being Saturday). No need having people try to work around you as you make these changes.

And that’s your Screen schemes standardized! When you are meeting with the teams, they may not understand why you are doing this. Stick with it – they will see the benefits eventually, and wonder why it wasn’t always like that.

But…uh…what about workflows?

Are you Forgetting something? Maybe you are,
Maybe you aren't - Are you Forgetting something? Maybe you are,
Maybe you aren't  Scumbag Brain

Yeah…workflows. I hadn’t forgotten about it – I was strategically putting it off. I promise!

My experience has been standardizing workflows is a bigger fight than screen schemes, which is why I chose to lead with that. People are willing to be flexible on fields – most people don’t use most custom fields anyways. But when you start messing with their workflows, you are messing with how they work. And yeah – that is part your job – but it’s going to be a fight. So start out with the screen schemes and build up some good will. That will make this easier.

With the Workflow schemes at least, you have the information at hand for each team. Teams are somewhat free-form in how they use fields, so you have to interview them first. But a workflow can only be used as designed. So start by looking at all the workflows for the projects you wish to standardize. Figure out the common points, and design a workflow (in your test instance!) that matches as many of them as possible.

Now that you have something designed, take it to those same teams. Explain to them what you hope to achieve, and demonstrate how the standard workflow you are proposing differs from the one that particular team is currently using. Go through the same thing with each team you did with the fields. Explain the benefits of everyone working on the same scheme. The object here is to build con-census from the ground up.

Look, I’m not going to sugarcoat it. People are going to disagree with you. Some are even going to do so passionately. Don’t make enemies of these people. If they care about how JIRA is being used THAT much, they are a person to have on your side. Figure out what their objections are, and figure out a way to address or otherwise work-around those objections. But show you are willing to work with their concerns, and they will be more willing to work with you.

Once you have all the concerns addressed and con-census built, schedule a change window and put it in place. You can do it manually, or by a plugin. Honestly, I keep harping on the plugin as an option as it eliminates some possibilities of you making a mistake. But if you are going to do it manually, at least export the workflow and re-import it into production, so you don’t have to worry about a mistake there.

And that’s it!

Look, this won’t be the easiest thing to do as a JIRA admin. People can be very passionate about how they work, and if you are seen as messing with that, they can get upset. But remember that they are upset because they do care, so if you can show that you are willing to listen and work with them, it should help.

So I know everything else in the world has been going a little crazy right now. I really hope you all survived the transition and ramping up your teams to work remotely. I’ve been stuck at home anyways while I recover from the surgery, but even before that I’ve been a remote employee for some time, so it still business as normal for me. I can write a post on tips and tricks from working from home, but it’s all likely advice you’ve heard elsewhere already, so unless requested I don’t have a plan to write on that topic. But if you are interested, let me know!

Don’t forget you can subscribe to receive new posts from this blog via email! To subscribe, just put your email in the form below this post! We also have an officially Unofficial Atlassian Discord chat. Swing in, ask questions, and see what’s going on! https://discord.gg/mXuRsVu

But until next time, this is Rodney, asking “Have you updated your JIRA issues today?”

Spring Cleaning for Custom Fields

Well, I’m back…or at least upright. Thank you all for the well wishes I’ve received. As much as I hate not posting an actual article, I really did need the week off to recover. But now it’s time to get back to it. As I’ve stated, this week’s article was going to be a reader selected topic. It seems there was some demand for each, but the clear winner was…

I’ll be tackling the other articles in the coming weeks, but lets look at Getting those custom fields under control this week.

A bit of a confession here.

First, let me say that as Admins, I think each one of us has various strengths and weaknesses. Me, for example. My obvious strength is Systems. I can tune almost any system to run smoothly. I can learn new back-end technologies with almost no lead-time and still successfully deploy.

However, my weakness as a JIRA Admin is I’ve never been able to execute a successful Cleanup. I’ve known the problem, and I’ve worked to fix it. But I’d rather be a con-census builder than a dictator when it comes to JIRA, and people LOVE their custom fields.

To manage this, I’ve always put safeguards to keep things getting worse. I re-use fields rather than create new ones, and really analyze whether new fields are needed. But on systems I’ve taken over, it’s rare I get a group to give up a field that was already existing.

That being said, if you want to learn from a real master of this topic, I’d highly recommend Rachel Wright’s JIRA Strategy Admin Workbook.

However, I’m going to give you tools to analyze the problem, see what fields aren’t being used, and give you the data to say to people “This…this is a problem we can fix.” Then I’ll show you how to go about cleaning things up. So, lets learn together!

Why does this even matter?

So, you might be asking yourself “JIRA is all about the custom fields, why should I worry about how many there are?” That, honestly, is a fair question. But the reason you should care is performance. It’s just a fact: The more custom fields your system has to parse through, the slower things like creating an issue, searching, etc will be.

Average response times per action (in milliseconds): custom fields test. Courtesy Atlassian.

Take a look at this chart above. This is from some testing Atlassian did. If we look at JQL Searching alone, at 1400 custom fields, the performance is abysmal at almost 3 seconds. However, doubling the custom field count to 2800 triples that number to nearly 9 seconds.

And yes, this is an extreme case. But users will notice a delay of half a second. You will get comments saying JIRA is slow even at that number, trust me. And it doesn’t take too terribly many custom fields to get searching to be that slow. That is why your custom field count matters.

Analysis

It is said the first step to fixing a problem is admitting you have a problem. Check. The second step it would seem is to figure out how big the problem is. Unfortunately, for this part we are going to have to go directly to the database.

For all my databases, I like to keep an administrative read-only account that will allow me to go in and query from time to time. This helps with making sure things are running properly, or diagnosing problems every now and again. Just make sure it’s read only so you don’t mess with something that aught not to be messed with.

Anyways, Atlassian has some queries that will help you figure out what fields are being used in your system. You can find all of them available here:

All the queries below I have confirmed are still working for MySQL 5.7 and JIRA 8.5.3.

Unused custom fields

select count(*), customfield.id, customfield.cfname, customfield.description
from customfield left join customfieldvalue on customfield.id = customfieldvalue.customfield
where customfieldvalue.stringvalue is null 
and customfieldvalue.numbervalue is null 
and customfieldvalue.textvalue is null
and customfieldvalue.datevalue is null
and customfield.CUSTOMFIELDTYPEKEY not like '%com.atlassian.servicedesk%'
and customfield.CUSTOMFIELDTYPEKEY not like '%com.pyxis.greenhopper%'
and customfield.CUSTOMFIELDTYPEKEY not like '%com.atlassian.bonfire%'
and customfield.CUSTOMFIELDTYPEKEY not like '%com.atlassian.jpo%'
group by customfield.id, customfield.cfname, customfield.description;

So, the first logical question is to ask is what fields are just not being used – period. These can probably be thrown out easily enough. Just a word of caution with this query: It can return fields that are being used by Apps using their own datastore. I’ve modified it to at least throw out results from JIRA Service Desk, JIRA Software, Capture, and Portfolio So make sure you are vetting this list for such false postitives. But if it’s a field you or your team has created, and it’s on this list, it might be time to have a discussion on whether it needs to be around.

Custom fields that have not been updated after date (YYYY-MM-DD)

select field.id, field.cfname from customfield field where field.cfname not in (
select item.field from changeitem item
JOIN changegroup cgroup ON item.groupid=cgroup.id
where item.fieldtype='custom' and cgroup.created > 'YYYY-MM-DD'
) and customfieldtypekey not like '%com.pyxis.greenhopper%'
and customfieldtypekey not like '%com.atlassian.servicedesk%'
and customfieldtypekey not like '%com.atlassian.bonfire%'
and customfieldtypekey not like '%com.atlassian.jpo%'

So this one returns fields that haven’t been used after a date. You’ll have to modify the query by replaying “YYYY-MM-DD” with a date you are interested in, but this one is handy for an end of year review, as it can answer “What fields have we not used this year”. If your teams haven’t touched the field in that long, you have a strong case for it’s need to go away.

Custom fields with low usage

select count(*), customfield.id, customfield.cfname, customfield.description
from customfield left join customfieldvalue on customfield.id = customfieldvalue.customfield
where customfield.CUSTOMFIELDTYPEKEY not like '%com.atlassian.servicedesk%'
and customfield.CUSTOMFIELDTYPEKEY not like '%com.pyxis.greenhopper%'
and customfield.CUSTOMFIELDTYPEKEY not like '%com.atlassian.bonfire%'
and customfield.CUSTOMFIELDTYPEKEY not like '%com.atlassian.jpo%'
and customfieldvalue.ID is not NULL
group by customfield.id having count(*) < 5
order by count(*) desc

Unlike the first query, this one will return fields that do not meet a certain threshold of usage. As written, this threshold is 5 issues, but you can adjust it by changing the number in “count(*) < 5”. As a rule, I like to keep this to one-tenth of one percent of the total number of issues in your instance.. So if you have 400,000 issues in your instance, the threshold would be 400. But that is a personal preference, you should set this wherever it makes sense for you.

So, what now?

You know have your information, and you know what fields are not being used. You can just go and delete them, right?

Well…

Not really. You should do some more digging. Who asked for that field to be added? What was their business case? Was that field just recently added? For those with a small usage, are they only being used in a specific project? If so, what is the usage within that project?

Also, use Botron Power Admin to see where all the field is being used (or not used). This might tell you who you need to talk to as well.

This should help you narrow down your candidates further. Once you do that, go back to whoever requested those fields. Take the data from your queries, and let them know that the fields aren’t being used, and are on a list to be removed. You can also include documents on how custom fields impact performance for everyone. If you are lucky, they’ll agree with the data and you can remove it. If not, start negotiating. See why they think they need a field they aren’t using. See if there is a way you can give them their ask without the field.

Custom Field optimizer (Data Center)

For those of you on Data Center, JIRA has a built-in Custom field optimizer you can use. It will go through and find fields that are not used often, or otherwise “misconfigured” (As defined by Atlassian).

As you can see, it’s alerting on a field in my instance that has a global context. This was one I specifically created to test the optimizer, so it’s Ideally, you should limit your custom fields to only be available to the projects that need it. So you can use this tool to find those with global contexts and if necessary, fix them.

Merging Fields (App Required)

So, lets say you have a duplicate field. How do you copy the information from one to another so you can delete one without losing data?

Well, as you can guess, this isn’t something that is able to be done in vanilla JIRA – so we have to turn to the Atlassian Marketplace – and specifically Adaptavist’s Scriptrunner plugin.

Now my groovy – it needs some work. But Scriptrunner has a number of built in scripts, including one to “Copy custom field values”. This does what it says on the tin. It takes the value in a source field, and places it into a destination field, overwriting what was there. For most cases, this won’t be a problem for reasons I’ll talk about later, but do keep this in mind.

To use this, you will first need to save a filter that will limit what issues it will copy the fields on. I use something along the lines of:

"Custom Field A" is not EMPTY

This will return all issues that has the custom field set. You can add another clause to weed out those that have the destination field already set, and handle combining those manually. In most cases, each of the duplicate fields would have only been used on a project or two a piece, and have no overlap, so this shouldn’t be too much of a concern, but as always do your research first and make sure that is not the case in your situation.

Now, I shouldn’t have to say this, but always test any changes you want to make before doing it in production. That’s very true here. This built-in script isn’t perfect, and you may have to pick it apart and modify it a bit to get to your use case. The only way to know if this is required is to do your testing.

That being said, I asked my colleagues at Coyote Creek about merging fields, and Neil Taylor gave me this as a solutions he’s used before. From testing it, it’s the easiest to implement and execute that I’ve found, and should handle most of your use cases.

So time to get cleaning!

So you know how to find fields that are good candidates to removal, and have an action plan to migrate the data should they have anything worth keeping in them. What are you waiting for? Get out there, start conversations, and get yourself on a better path to better JIRA Performance.

Don’t forget to subscribe to receive new posts by email. It’s the easiest way to be notified as soon as the posts go live. You can sign up with the form at the bottom of the post. And don’t forget to check out our Atlassian Discord Chat!It’s always interesting to see who has popped up there and what’s being discussed. https://discord.gg/mXuRsVu

But until next time, this is Rodney, asking “Have you updated your JIRA issues today?”

A JIRA Guy’s New Years Resolutions

HAPPY NEW YEARS! I hope you had a good 2019 and look forward to a great 2020!

As you know, 2019 was at times….rocky, but all in all it wasn’t terrible. This blog has been one of the bright spots, actually! Hearing from readers out there about how much they are getting from the blog really just makes my day every time!

Where I live, it is a tradition that we resolve to improve at least one aspect about ourselves over the new year. So, that got me thinking, as JIRA Admins, what can we resolve to improve about how we work and how we run our instances. So without further adieu, here we go!

I resolve to not make any config changes without notifying all users first.

This one is easier said than done. But this one will save you so much headache in avoiding unexpected impacts from a change.

The challenge here is there are a lot of changes that you think will have no impact on user experience, and it’s tempting to just rush in and change it. Especially when explaining what you are changing and why feels like writing a light novel. I know, I’ve been guilty here too!

But giving yourself a chance to pause to explain what you are doing and why you are doing it will give both you and your users some perspective. If you are having trouble stating why you are making a change in a way that normal users can understand, do you really understand the change you are about to make? And if that’s the case, why are you making a change you don’t fully understand?

Now another thing here is this requires some lead time. If you send out a rushed email than immediately make a change, you haven’t really notified anyone. It’s the giving it time for users to digest the information and send in the feedback that will show the true power of this resolution.

I resolve to clean up Custom Fields from JIRA.

I see this one all too often. JIRA went through a “wild west” period where either a) all user requests for new fields were automatically stamped “Approved”, or b) everyone had admin rights and just added fields without a care in the world. It’s nothing to be ashamed off, but it is something you should resolve to fix, as an over-abundance of custom fields can cause system performance problems.

Now I’m not going to lie, this is a political quagmire if ever there were one. Users LOVE their custom fields, especially Project Managers. So if it were me, I’d start with the low hanging fruit. In this case, that would be duplicate fields. This one is easy for users to see why it can be a problem. Try doing a JQL search on a field that shares it’s name with another. Which field is which? Which has the data you need? Who knows!?

This does lead to the problem of what to do with the information contained in whichever copy you are getting rid of. Not going to lie, that is not an easy question to answer in Vanilla JIRA, and rightfully deserves a post all it’s own. However, the ever helpful folks at Atlassian Community has a few pointers to get you started.

If you manage to clean up those fields, then comes the next easiest category: Those that are “close” to each other in purpose or meaning. For example, lets say you have the following list of fields:

  • Dev Customer
  • Customer
  • Customer (Accounting)
  • UX Customer
  • Customer Number
  • Customer ID

You now have six fields, which all contain essentially the same kind of information. In an ideal case, any custom field you create should be “generic” – that is to say it should be general enough you can drop it in any project where it’s needed and it makes sense. In this case, I’d work with five of these groups and migrate them all to use the “Customer” field. It’s generic enough that the meaning can be used for all of these applications, and gets rid of five fields in one go.

I resolve to gain and/or regain control of my JIRA instance.

So this one isn’t as universally applicable, but for those of you dealing with it, it can be a hassle. You do the work, get things cleaned up in JIRA and the system working smoothly, just to have some guy – who lets be honest should NOT have JIRA Admin rights – come in, create 20 new projects with 40 new fields.

It’s a story that does happen. That is why you should strive to regain control over your instance, and be the gate-keeper for any such changes. Do you really need 20 projects, or because they are all short lived, could you do with one and utilize Versions/Releases to organize that work?

Now, I’m not going to say this will be easy. Anybody with JIRA Admin rights that aren’t you might have some clout – or at least the ability to complain very loudly to the right people. You will need to defend your decision to take away their rights by showing how the changes they are making are not following best practices, and are actually having a detrimental affect on the instance. But remember, YOU are the JIRA Admin. The health of the instance is ultimately your job, so state that clearly. And remind them that you are open to requests, but you will guide the discussion to something that is better for both the users and the platform.

I resolve to make myself as much of a priority as my systems.

I get it. Your teams and users depend on you to keep the system running and updated. You know what makes it easier to depend on you? Not burning out.

I’ll see it on reddit from time to time where a Sysadmin has passed away from a heart attack, or otherwise burned out and is switching careers. They are always the type who hardly ever took vacation, never took personal days, and felt like the weight of the company is on their shoulders. I’m telling you now – don’t be that guy. I mean I play around with these systems for fun in my spare time, and I still have to unplug every once and a while!

So if this sounds like you, just relax. And if you are at a company that would fire you for taking a few days off every now and again, you’re at the wrong company! Every company should be concerned about burn out, and if they aren’t that isn’t a healthy environment, and you should start looking now for a better year ahead.

Again, Happy New Years!

So, what are your JIRA Admin Resolutions? Are you doing one of the ones I listed here, or are you doing something different! Please let me know. Next week I’ll be reviewing a app (or plugin for us curmudgeons) that literally blew my mind when I saw what it could do. But until then, I hope you have a great start to 2020! I’m Rodney, asking, “Have you updated your JIRA Issues today tomorrow?” Happy New Years Everyone!

Getting started with JIRA Data Center: Pt 2 – Updating JIRA to JDC

So, when we left off as week, we had all the support system ready to go. As a review, we have the NFS Server, database, and Load balancer – each needed for JIRA Data Center to work to it’s fullest potential. Now we need to convert the actual JIRA instance into our first node for JIRA Data Center.

First, I want to address the elephant in the room. You may have noticed that this blog post is being released a little later than normal. That is because Atlassian released another vulnerability report today. This time its for Confluence Server and Data Center. I won’t be covering it today – I’d much rather get back to the JIRA Data Center install. But I’ll post it here for your own research

Also, as a cleanup note from last week, I forgot to mention one thing. All the systems you use should be configured to use the same NTP server for time synchronization. That is to say not the same *pool*, but the same *server*. What, me? Make that mistake? And only find it while writing this article? Never! </sarcasm>

Okay yeah, I did that. I am not perfect, and sometimes some things do slip by. But I don’t want you to make that same mistake, so I figured I’d own up to it now and not let you fall into the same problem.

Installing the JDC License

Before we do anything further, we need to install a JDC license key into JIRA. That way once we get things done and bring it online, you won’t have any issues.

To do this, we need to find our Server ID. This can be found by going to System -> System Support, then looking for “Server ID” on that page.

After you grab this, go to https://my.atlassian.com if you are getting a trial license. And honestly, I’d recommend you get a trial license while you are doing your initial testing. No need to put money down while you are doing a proof of concept.

Click on the “New Trial License”, then on the next screen select your JIRA flavor from the drop down (Either JIRA Software or JIRA Service Desk). Then once the rest pops up, select Data Center. You will then enter your Server ID we just grabbed.

This will generate a trial license that lasts 30 days. You can renew this a number of times, but it’s not infinite, so use this time wisely.

Copy your new trial license, then in your JIRA instance, go to Application->Versions and licenses, and paste it in. Then you are done with this step, and we can begin making JIRA into a proper node.

Moving the Shared Home.

From here on out, we are going to be following the doc “Installing JIRA Data Center”

Per the guide, we are setting up our shared. First, shutdown your JIRA instance.

systemctl stop jira

Once it has stopped, navigate to your JIRA Home Directory, and copy the following folders to your shared NFS directory.

  • data
  • plugins
  • logos
  • import
  • export
  • caches

Once copied, be sure to change the owner back to jira so that it can work once you bring the system back up.

chown -R jira:jira /data/jira/sharedhome

And this part is done! One step closer to having JDC up and running.

Setup your cluster.properties file

Now that we have the shared home setup, we need to setup the file that tells JIRA that a) It’s a JDC Node, and b) where to find the rest of it’s files. This is the cluster.properties folder, and it’s placed in the root of the local home folder.

This file will have the following settings:

# This ID must be unique across the cluster
jira.node.id = jnode1
# The location of the shared home directory for all Jira nodes
jira.shared.home = /data/jira/sharedhome

As a best practice, I like to put the system hostnames into the node id, that way we know where to look when we are having issues, but honestly it’s arbitrary, so use whatever works. The only requirement is that each node has a unique name.

There are some more options you have in the cluster.properties file, but unless you are having problems, you shouldn’t need to use them. But, I want you to at least be aware of them.

As a last setting, if you are running on linux (like we are), go to your <jira_install>/bin/setenv.sh file, and add the following line.

ulimit -n 16384

Now we need to unblock two new ports that JIRA will use to communicate between nodes.

firewall-cmd --permanent --zone=public --add-port=40001/tcp
firewall-cmd --permanent --zone=public --add-port=40011/tcp
firwall-cmd --reload

At this point we can start JIRA as a data center instance!

systemctl start jira

If all goes well, when JIRA will come up as a single node Data Center instance.

Standing up additional nodes.

Now, we can really take advantage of the super power of JDC: Adding nodes!

Now, if we want to go easy mode, we could just clone the VM and modify the cluster.properties file. But we are JIRA Guys, we don’t do easy mode here!

From a fresh install, we need to install the same version of JIRA that is on our first node. After this install, we need to see if we are using the same userid and group id for JIRA. If not, it will cause problems with NFS. So run the following on all nodes:

id jira

If any differs, it needs to be changed to match the existing nodes. Note this also means you will need to change the file permissions that belong to that particular JIRA user on that node. I could go through the guide on how to do this, but someone else has already done a much better job than I can.

With that minor annoyance out of the way, we can copy over the local home folder. The best way I found to do this is to tarball the entire thing up and scp it to the new node.

tar -czvf jira_home.tar.gz <jira_local_home>
scp ./jira_home.tar.gz <new_node>:~/

Once on the new node, unpack the tarball and move the home folder to the appropriate place. If you used default locations for both nodes, you can probably just unpack it at the root folder and it will find it’s way to the right place. Then we just make sure the permissions weren’t broken:

chown -R jira:jira <jira_local_home>

Next we modify the cluster.properties folder, changing the jira.node.id to something unique in the cluster. As an added note, if your nodes do not resolve to DNS, you will need to add the following to this file.

ehcache.listener.hostname = "<ip_addr_of_node>"

You will also need to do this for the existing nodes as well. I also like to take the extra step of adding the file share, database, load balancer, and all nodes to the hosts file of all systems involved. That way even if you do have a DNS Failure, your cluster won’t die. Host entry take the following format:

<ip_addr>    <fqdn>  <alternate_host_name>

An ounce of prevention is worth a pound of cure, especially when IT decides to move up an upgrade of a DNS node without telling anyone. Yes, that’s a true story!

At this point, you should make sure that the system has the shared home mounted. You can do this by copying the entry for the NFS share from the first node’s /etc/fstab file and placing it into the second. Then make sure the folder structure up to the mount point is present. You can uses the -p flag on mkdir to make all required folders in one command.

mkdir -p /data/jira/sharedhome

You can now mount the share with the mount command.

mount /data/jira/sharedhome

Take a second and test that the JIRA Account can write and see all files in there. Now unblock all the correct ports you need for JDC:

firewall-cmd --permanent --zone=public --add-port=8080/tcp
firewall-cmd --permanent --zone=public --add-port=40001/tcp
firewall-cmd --permanent --zone=public --add-port=40011/tcp
firwall-cmd --reload

If everything looks good, start up JIRA as you would on any other system, and follow the logs. IF all goes well, you should be able to access the node by going directly to the IP Address for it.

At this point, check the copyright info at the bottom. You should see the new node name at the bottom after the version info:

Log in, and check the Health Check, if all is good you should see green check marks on the Cluster.

Now the only thing to do is add it to your haproxy config, restart haproxy on your load balancer, and start sending traffic to your new node!

And you made it!

It’s an involved process, I tried not to lie about that. But it’s not an Everest sized challenge. You might have to get a bit outside your comfort zone and play with some technologies that as a JIRA Admin, you haven’t played with before. But my view on it is in order for us to grow as Admins, sometimes we need to step outside our comfort zones. But at least you have a road map to help guide you on your Data Center Migration!

I’m still looking to do a Lightning round Q&A session next week, so get your questions in. Anything from how I do testing and capture images for the blog, to how do you deal with a particular challenge, to what kind of server(s) do I run, it’s all on the table.

Also have an article I’m excited to write up for the New Years – which incidentally enough, is a blog day.

So until next time, my name is Rodney, asking “Have you updated your JIRA Issues today?”

Following the Breadcrumbs: Decoding JIRA’s Logs

Well, that’s a thing. For those that don’t know, Rachel pretty much wrote the book on being a JIRA Admin. So, I guess it’s time to put up or shut up. JIRA Logging, lets do this.

For those of us in the trenches, logs are life savers. No, really. When something goes wrong within an application, logs often are our first, and sometimes only view into why it went wrong. You can tell a lot about what JIRA is doing from it’s logs – If you know what to look for.

As always, we start any look into a new aspect of JIRA by looking at the documentation. Here, we find the relevant document under Logging and Profiling:

Where to find your logs.

Finding your logs is actually pretty simple, assuming you know where your home directory is. Your log files for JIRA can be found at <jira_home_directory>/log/.

Navigating here, you will see several log files, usually along the lines of “atlassian-jira-<name>.log. If that log has had a roll-over, you will see some repeats of them, with older ones being followed by a number.

The one we will be interested in nine times out of ten is the atlassian-jira.log file.

But what about the Catalina.out file?

You my have noticed another log file in the JIRA install directory called catalina.out. This is tomcat’s log file, and while it replicates most of what is in atlassian-jira.log, it won’t have everything JIRA’s doing in there. Furthermore, it will muddy the water with additional information from Tomcat. If you haven’t changed any settings in the JIRA install directory, you won’t have a lot of reasons to go into there.

However, I have found catalina.out helpful when I’ve had an issue starting JIRA because the JVM environment was misconfigured. In that case, JIRA’s logs will have nothing, but Tomcat will report the Java error. If your JIRA is up and running though, don’t worry about this log.

Typical Logs

So, here is a snapshot of the logs after doing some typical things within JIRA:

Yeah, you may have to zoom in a bit there. But for the whole, looks like organized chaos. And well, it is. And in a live production environment, these may be zooming by so fast you may not have a chance to actually absorb what’s happening. With a bit of practice you can get to a point where you can watch the logs stream by and see only the things that look out of the ordinary…but even that’s not perfect. However, lets look at one entry, when I access the login page for JIRA.

2019-10-29 08:08:44,423 http-nio-8080-exec-23 INFO anonymous 488x35978x1 xmd6sf 192.168.<redacted>,192.168.<redacted> /rest/gadget/1.0/login [c.a.jira.index.MonitoringIndexWriter] [lucene-stats] flush stats: snapshotCount=6, totalCount=63, periodSec=486768, flushIntervalMillis=81128149, indexDirectory=null, indexWriterId=com.atlassian.jira.index.MonitoringIndexWriter@19cc4430, indexDirectoryId=RAMDirectory@5d7db2f9 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@14d78abf

So, what do we have going on here. At first glance, we notice several parts.

Time Stamp and Misc Information

First, we have the time stamp. This is often the best thing to filter off of. If you know when an event you are interested in happened, you can start around that time and work your way backwards. It can narrow hundreds of thousands of lines to maybe a few hundred.

The next bit is just an identifier that tells you what part of JIRA is generating the log. for the above entry, it’s the web server. It may take some digging to figure out what is a part of what and honestly unless you are doing log aggregation, it likely isn’t going to help you a whole lot.

Logging Level

The third bit – that’s interesting. It’s the logging level. Logging levels tell the application how much of what’s going on you want recorded, and come in five levels:

You may notice I put them onto a scale. That is because as you change the logging level, you are increasing or decreasing how much of what is going on is recorded to the logs. I could do a whole post on logging levels alone, but the gist of it is if you set debug, you can almost watch all the variables change in real time in the logs. If you set it to Fatal, JIRA will only log entries when something catastrophic happens, and even then will likely leave out details leading up to the crash.

So this leads to the question: “If not enough information is a bad thing, why don’t we set everything to Debug and log all the things?”

Bad Admin! No Treat!

If you haven’t guessed, there is a trade-off. Here, it’s disk space. If you set everything to Debug before you leave on Friday, by the time you are sitting down for dinner on Saturday, you’re going to be paged for a full disk. No really, you’ll be surprised how fast text can fill up a hard drive.

Speaking in general, you can also use Logging levels to tell if something bad is happening. The Developers will assign different events to different logging levels, based on their best guess on how important it is to log. That being said, I’ve seen items that were Warn or Error that were no big deal, and I’ve seen Info level log entries that were actually concerns. The easiest example I can think of is actually in my logs right now:

2019-10-29 08:30:11,656 Caesium-1-1 INFO ServiceRunner     [c.a.crowd.directory.DbCachingRemoteDirectory] Incremental synchronisation for directory [ 10000 ] was not completed, falling back to a full synchronisation
2019-10-29 08:30:11,656 Caesium-1-1 INFO ServiceRunner     [c.a.crowd.directory.DbCachingRemoteDirectory] INCREMENTAL synchronisation for directory [ 10000 ] was not successful, attempting FULL

This is a log level INFO, but it’s saying something causing it to fail an incremental synchronization from my LDAP Directory. In my test environment, the LDAP directory is small enough this won’t be a problem, but in a full corporate LDAP directory, this can cause a significant overhead. I’ve even DDoS’ed my own JIRA instance trying to do a full directory sync. And again, this is only a “INFO” level log…

Identifier

After the Log Level, we have the identifier information. For web traffic and actions, this will tell you whose account initiated the action that resulted in the log. If you have a problem that only one user is reporting, this can help as you can search the log for that username. Of if you want to know if that person did something that caused problems, there you go too.

For web traffic, this is also accompanied by a couple of IP Addresses, usually the user’s clientside IP Address, followed by the IP address of your proxy. This can be helpful for identifying traffic originating from outside your network, or investigating access that didn’t look right.

Package

After you have your identifier, you will see a section in a bracket. This is the package that generated the log. In lay mans terms, this is a more specific part of JIRA that generated the log. This can be a specific plugin (or part of a plugin), or a particular routine within JIRA. You can see the list of packages and their associated logging levels within the System -> Logging and Profiling section of the Administrative Control Panel in JIRA.

In speaking of Logging and Profiling, you can also change the logging level per package in this section as well. These changes will persist until the next time you restart JIRA. In a future post we’ll also cover how to change them permanently. You also have access to an even deeper logging level here called “Trace” – but it is recommend you only use that if you are asked to by Atlassian Support in order to help with a specific issue. Your disks will thank you.

Message

After all these details, we get to the meat of the log – the message. This is what happened, or didn’t happen, that warrants logging. This is the “What” to the rest of the logs “Who, When, and Where”. This is where you can dig in to find whatever problem you are looking for.

And that’s what’s in your JIRA Log

And that’s pretty much it. I do have one more thing for you. For me, reading white logs on a black background can make things easy to get mixed up. To help with this, I wrote up a quick little script that will look up where you JIRA Log is, and pass it through an Awk filter so that it will colorize things based off the log level for each entry. It will also follow the log, which means it will display entries in real time as they are updated into the logs, and keep doing so until you hit “Ctrl+C”.

For reference, the colors each log level will be:

  • DEBUG: Green
  • INFO: Default
  • WARN: Yellow
  • ERROR: Red
  • FATAL: Magenta
#!/bin/bash
# Rodney Nissen
# Version 3
# Updated 10/29/2019
# Update JIRAINST to point to your JIRA Install Directory

JIRAINST="/opt/atlassian/jira"
JIRAHOME="$(sudo cat $JIRAINST/atlassian-jira/WEB-INF/classes/jira-application.properties | grep "jira.home = " | cut -d ' ' -f 3 | tr -d '\r')"
JIRALOG="$JIRAHOME/log/atlassian-jira.log"

sudo tail -f -n 50 $JIRALOG | awk '
  /DEBUG/ {print "\033[32m" $0 "\033[39m";next}
  /WARN/ {print "\033[033m" $0 "\033[39m";next}
  /ERROR/ {print "\033[31m" $0 "\033[39m";next}
  /FATAL/ {print "\033[35m" $0 "\033[39m";next}  
  1
'

I hope you will get as much use out of this tool as I have. As I stated in several parts of this post, I think this is the kind of subject that deserves a series of posts, not just one. As such, I’m thinking I’m going to do at least two more – one is an in depth dive into Logging Levels and what they actually mean, and another on how to play around with your logs. But until then, I’m Rodney, asking “Have you updated your JIRA Issues today?”

Why you should care about Enterprise Releases

So, confession time. I normally like to have a post written in advance of a release so that I have time to go back to it with fresh eyes and edit it. This week however, I haven’t been able to figure out what to post. I originally thought about doing something around dressing up your instance for Halloween, but then I remembered something very, VERY important. I’m going to be brutally honest with both myself and you for a moment…

I am VERY terrible at web design…

This is perfectly alright. When I go to brand (or rebrand) a JIRA instance, I’ll usually reach out to Marketing or the website team, coordinate with them on color palette, and try out several variations. I’ve even been known to mock up a few palettes, put together a poll, and put it to a user vote. Your users have to use this platform every day. They should have some say in how it looks.

BUT – maybe this is not a lesson I’m qualified to write up…at least not yet. But this left me a day away from post time and no article. Then, while downloading a new version of JIRA for a test I was running, I noticed this:

A new JIRA release – but not just any kind of release – but an Enterprise Version.

So, why should I care about an Enterprise Release?

To understand this, first we need a bit of a history lesson.

While at VMware, we were fortunate enough to have both Atlassian TAMs and Premiere Support from Atlassian. I cannot speak highly enough of these teams – without them and learning from them, I don’t think I’d have gotten as far as I have as an Atlassian Admin.

One of the key benefits of Premiere Support is something called a Health Check. You would be able to submit your Support Zip to the team at Atlassian, and they’d take a look at it and see if there are any changes or tuning adjustments you can make to improve performance. This quickly allowed our instances to become some of the best maintained and performing instances inside VMware.

It became our policy to submit a health check prior to any upgrade we planned to make sure we were on a good footing. Often during our health check, Premiere Support and our TAMs would advise us of what they considered “Blessed” versions. That is, versions of their software that had undergone testing by Premiere Support and were considered stable enough to recommend to large enterprise clients.

Shortly before I left VMware, this concept morphed into an Enterprise version. This version was something of Atlassian’s take on Long Term Support versions. The plan is that any bugs will be backported to this Version.

What do you mean? Isn’t a version a version?

Eh – this gets sticky if you haven’t worked in software.

Software versions are something of a code. Lets take Atlassian’s releases. They often are encoded with three numbers separated by a period. For example: 8.5.0

The first number, Eight, in this is the Major or API Version number. In most software companies, they don’t increment this unless they have made major architectural changes – in lay mans terms that is to say they pretty much rewrote major parts of the code. That is why going from v1 to v2 is a big deal for them.

For Atlassian, a change in this number means they have made changes to either the underlying JAVA API or the REST API. When this changes is when I start worrying whether plugins or integrations will break. They can break on other releases, sure, but they are more likely to break when this changes.

The second number, five, is Feature Release Number. When this changes is usually when Atlassian will add new features to their products. This can sometimes either break or render obsolete plugins, but as I said that is more likely when the API Version changes. This is typically what I mean by version.

The third and final number, zero, is the Bug fix number. This is when Atlassian will fix bugs in their products, but add no new functionality. This is where the magic in Enterprise Releases comes in. For Enterprise releases, any bug fixes Atlassian makes for future versions that also affect the Enterprise Release, Atlassian will back port to the Enterprise Release. That is why most Atlassian releases will only get to a bug fix number of 3 or 4, but Enterprise releases will get to a bug fix number of 10 or 11.

This also gets to my advice about never installing software ending in Zero. These versions have not had a bug fix release yet, which means they are statistically more likely to have bugs than other versions. Don’t get me wrong, ALL software has bugs, but at least those not ending in zero have had some fixed.

Bug fix releases are also *generally* safer upgrades than Feature Releases or API Releases. That is to say, going from 8.5.0 to 8.5.1 is *generally* safer than going from 8.4.x to 8.5.0 or 7.6.6 to 8.5.0. HOWEVER, never trust things as infallible, and always test your upgrades in a pre-production environment.

But I’ve seen Atlassian back port security bugs to non Enterprise Releases? What’s up with that?

Yes, this is true. Some bugs are so insidious, so dangerous to security that Atlassian will do the work to back port them to as many versions as are still supported. You should always pay attention to Security Advisories and plan your upgrade cycles accordingly. But that being said, for an Enterprise Release you’re also going to get other bug fixes on top of the security ones, so why not go with that?

Is that the only benefit?

Not. Even. Close. Atlassian will also do performance testing on their Enterprise Releases, and tell you the result.

This can help you in your decisions around scaling and whether an upgrade is worth it. You might even be able to use this to sell the upgrade to your user base. I have yet to meet the user who didn’t appreciate a faster JIRA.

They also prepare an REST API Change log so if you are hopping from Enterprise Release to Enterprise Release, you can prepare your integrations for the upgrade that much easier

Also, to prevent you from going page by page to collect all the changes from previous versions, they also do a full change log between Enterprise releases here.

So, yeah, Enterprise Releases. They really are a good thing.

Every upgrade is a decision. Sometimes you have to have a new feature or bug fix and cannot wait until they next Enterprise Release to get it. It happens – I’ve been there. That being said, for the long term support you get, if you are limited on upgrade cycles, you don’t have a lot to lose by going with an Enterprise Release. So it’s definitely something to consider as you are going through the release notes to figure out a target version.

So for next week I’m still not sure what to write about either. However, I did have an interesting challenge on LinkedIn from last week’s post that I’m honestly considering…

That might be interesting. I still have the VM I did that on, so it might be a thing. Until then, My name is Rodney, asking “Having you updated your JIRA issues today?”

Installing JIRA from scratch, Part 3: Reverse Proxies and Final Tweaks

So welcome back to the third (and final) post about installing JIRA from a clean linux setup. Last week we deviated a bit from this series, but there was a reason. I needed the extra time to study, and well, this is going to be a long post.

Study for what you say? I’m glad you asked! As of last Thursday, I can officially add this shiny thing to my resume:

So shiny…

I also took the opportunity to get my Scaling JIRA badge (ASB-SCJ) to extend my ACP-JA. But I did pass the test! Needless to say, this is an actual video me last week.

But with self-congratulations out of the way, I think it’s time we continue on with the series!

What were we working with again?

So, as it’s been a few weeks, lets take a look at where we are. We started with a clean CentOS install on a VM with the following specs:

  • OS: CentOS 7 Linux
  • RAM: 4 GB
  • Processors: 2
  • Root Hard drive of 30 GB
  • Extra Hard drive of 50 GB to house the database and JIRA’s home directory

On top of this, we installed MySQL 5.7 and JIRA Software 8.1.3, and had everything in a working state, but running off of port 8080.

However…Security Updates…

Fun fact, did you know that before I started with my current job, I had about three weeks of posts saved up to allow me to focus on my career for a bit? That means I had the Part 2 for this series finished and in the can about two weeks before it was published. And honestly, I had done the install and captured the pictures weeks before that. Which leads me to the reason I bring this up…

On Sept. 18, I got a nice message from Atlassian, letting me know about a security vulnerability. As the version I installed was not covered under this or any previous alert, I opted not to change my post to address it – opting instead to disclose the fact now. However, I do *highly* recommend you check back with this page regularly, and plan your upgrades and installs accordingly.

So why do we need a reverse proxy?

So, for most seasoned people, this will be obvious. But I’m going to cover it because it’s always helpful to explain the why of what we are doing. Especially when it seems like an extra step.

When Tim Berners-Lee created the original http protocol, he did not imagine a world where we’d be handling everything through websites. Honestly, he was just making a mechanism to share public research papers. Being public, he didn’t foresee the need for security.

The end goal of all this is to say, if you aren’t specifically using https (<-note the “S” at the end), every time you or someone logs into your JIRA instance, you are sending that password in clear text to the server. Sounds bad? It’s because it is.

The HTTPS protocol is how we fix this. It takes the web traffic you send, encrypts it in such a way that only the server can decrypt it, and sends it on to the server.

JIRA’s tomcat can do this encryption/decryption with some work and time – however, it’s not great at it. Combine that with the fact that it will use resources in your JVM heap that could go to JIRA, it’s generally agreed that this isn’t the best solution.

So we setup what’s called a “Reverse Proxy”. It sits in front of the web traffic, and handles the encryption/decryption much better than JIRA can. That is to say, JIRA will feel much more responsive while still staying secure.

Sounds Great, how do we get started?

First we need to ask: Which service we are using as a proxy? There are several options, with two being the ones most people choose: Nginx and Apache. Today I am going with Apache for two reasons.

The first, and likely most important for me, it’s what I know. I’ve been setting up some variation of LAMP servers for over 15 years now. It’s a system I know well by now.

Second, and likely most important for you: It’s the one I found documentation for on Atlassian’s knowledge base. When in doubt, follow the docs.

So…more reading material then?

You got it! In today’s reading corner, we have “Securing your Atlassian applications with Apache using SSL” from Atlassian. Thankfully it’s the only one we’ll need this week.

I’ll be deviating from it in a place or two (mainly in the Apache Configuration), but I will note where I do this. These changes are quality of life improvements, which makes my (and hopefully your) life easier.

SSL Certificates

If you are lucky, you will have someone at your company who can grant you access to – or otherwise get – an SSL Certificate for you. Unfortunately, you are the JIRA Administrator, so if you don’t have this resource, you will need to get one yourself.

You can get them from a variety of sources, but I recommend you stick with reputable Certificate Authorities. It would very much ruin your day if you went cheap just to find Mozilla or Google pulled that CA, causing your site to stop working across the internet.

For my home environment, I use a wildcard certificate from Comodo. No matter who you use, if they are reputable, they will have a knowledge base to help you through the process, so I won’t cover that at this time, instead assuming we are starting after you have received your certificate.

When I ordered mine last, I got it within the hour, so if you have to wait to get one, you won’t have long to wait.

So, what will I get when I finish

When you finish, you’ll likely get a zip or other compressed file containing, your certificate, several other certificates, and a key file. Save the key, it will look something like this.

Now you will notice my key file is missing. Please, for all that is good, keep your key secure at all times. With that file, anyone can decrypt the packets heading to your server.

Of what you can see here, we have three chain certificates, plus my certificate (Star_folden-nissen_com.crt). You will need to know what parts are what. Transfer all these files plus your key to your server’s filesystem, taking note where each is.

So, we’re finally going to do something on the server?

Yes! First thing first, we want to make sure Apache is installed on your system. Here we’ll use a variant on CentOS called httpd. This still very much is Apache, just with focus on only putting out the most important updates – in such a way that doesn’t break your configuration.

yum install httpd mod_ssl

If you need to install it, you’ll see the familiar confirmation. Type “y”, followed by enter to proceed.

Looking at the configuration directory, we should see the following files

cd /etc/httpd/;ls -lah

For the next section, we’ll be focusing in on the conf.d folder.

Configuring Apache

I told you when I’d be deviating from the Atlassian Documentation, and where it is. I like to separate each service a reverse proxy is supporting into it’s own conf file, as it makes everything easy to look through. While not expressly explained in the documentation, it doesn’t go against it, but I’d like to be clear about any deviations I make and why.

Moving into the conf.d folder, we’ll see the following files:

  • autoindex.conf
  • README
  • ssl.conf
  • userdir.conf
  • welcome.conf

Really, the only one we’ll care about today is ssl.conf, however we’ll want to disable the rest from apache. This can be done by renaming the extension of each from .conf to anything else. For example:

mv autoindex.conf autoindex.bck

After you run this for everything but ssl.conf, you should see this:

Now lets dive into ssl.conf. Open it up with the text editor of your choice (I prefer vim, but nano is admittedly much easier to learn). You will see a bunch of settings required for ssl. We’ll keep most of these, instead commenting out anything between <VirtualHost _default_:443> and </VirtualHost> (including these two lines).

After this, we are going to create a new file, “jira.conf”, and entire the following.

<VirtualHost *:80>
    ServerName <subdomain>.<domain>.com
    Redirect Permanent / https://<subdomain>.<domain>.com/
</VirtualHost>


<VirtualHost *:443>
    ServerName <subdomain>.<domain>.com

    ProxyRequests Off

    <Proxy *>
         Require all granted
    </Proxy>

    ProxyPass / http://<subdomain>.<domain>.com:<port>/
    ProxyPassReverse / http://<subdomain>.<domain>.com:<port>/

    SSLEngine On
    SSLCertificateFile /path/to/your/cert.pem
    SSLCertificateKeyFile /path/to/your/privkey.pem
    SSLCertificateChainFile /path/to/your/first_chain.pem
    SSLCertificateChainFile /path/to/your/second_chain.pem
    ...
    SSLCertificateChainFile /path/to/your/nth_chain.pem

</VirtualHost>

A few notes here. You need to put in as many SSLCertificateChainFile entries as you have chain files, but be sure to include them all. This example list the files in a .pem format, but in reality they can be any number of different file extensions, so don’t think you need to chain it to be a .pem specifically.

But what if I have my JIRA on a sub-directory?

This is supported, but we’ll need to make Apache aware of this fact. Assuming your sub-directory is <subdomain>.<domain>/<contextpath>, we’ll get

<VirtualHost *:80>
    ServerName <subdomain>.<domain>.com
    Redirect Permanent /<contextpath> https://<subdomain>.<domain>.com/<contextpath>
</VirtualHost>


<VirtualHost *:443>
    ServerName <subdomain>.<domain>.com

    ProxyRequests Off

    <Proxy *>
         Require all granted
    </Proxy>

    ProxyPass /<contextpath> http://<subdomain>.<domain>.com:<port>/<contextpath>
    ProxyPassReverse /<contextpath> http://<subdomain>.<domain>.com:<port>/<contextpath>

    SSLEngine On
    SSLCertificateFile /path/to/your/cert.pem
    SSLCertificateKeyFile /path/to/your/privkey.pem
    SSLCertificateChainFile /path/to/your/first_chain.pem
    SSLCertificateChainFile /path/to/your/second_chain.pem
    ...
    SSLCertificateChainFile /path/to/your/nth_chain.pem

</VirtualHost>

Please note the subdirectory here includes a leading slash, but not a trailing one. Be sure yours looks like that as well, otherwise you will have problems.

Also keep note of what you have as <contextpath> – we’ll need it again here in a bit.

So what now?

Before bringing Apache online, we’ll need to test the configuration.

apachectl configtest

Assuming everything is alright, you’ll see a “Syntax OK”. It won’t catch every problem, but it will catch some, which is better than nothing.

Next we’ll need to allow firewall traffic to ports 80 and 443:

firewall-cmd --zone=public --permanent --add-service=http
firewall-cmd --zone=public --permanent --add-service=https
firewall-cmd --reload;

And finally, we need to tell SELinux that Apache can be allowed to do an HTTP Relay.

setsebool httpd_can_network_relay on

After all of these steps are completed, we can start, enable and test our JIRA proxy

systemctl start httpd
systemctl status httpd
systemctl enable httpd

And we’re done?

If you’ve done everything correctly, browsing to your JIRA Server on port 80 should automatically redirect you to port 443 and show JIRA.

Note: if you are using a subdirectory, JIRA will not come up at all. But you should still see a secure connection. We’ll fix this in a later section.

Notice the notification bar across the bottom. This is because we need to make JIRA aware it is running through a proxy.

On the filesystem, navigate to <jira_install_dir>/conf/, and open up server.xml in your text editor of choice. You should see something like this.

Under the connections section, add the following without modifying anything else in there.

scheme="https" proxyName="<subdomain>.<domain>.com" proxyPort="443"

When done, the connector should look something like this:

Remember that context path from earlier?

If you are hosting your JIRA on a subdirectory, you will need to make JIRA aware of that here as well. Go down to the context section, and where it says path, enter “/<contextpath>”. As with earlier, note the leading slash, and do not put a trailing one. It should look something like this when you are done.

Anytime you make changes to server.xml, they will not be reflected until you restart JIRA.

systemctl restart jira;tail -f <jira_home_dir>/atlassian/application-data/jira/log/atlassian-jira.log

This should restart JIRA if you have installed it as a service, and then bring up the log file for you to follow as it boots. Anytime you make changes to configuration files, it’s always a good idea to monitor the logs during startup – just to make sure the system comes up cleanly.

SO ARE WE DONE NOW!?!?!?!?!?

Uh…no. We have two last things to do. If we go to the site now, we no longer see that notification bar. HOWEVER, if we log into our admin account, we see the following message.

Here, we simply click “Update Jira’s base URL” and it should update to the URL you are accessing it from.

While we here, we also want to run a health check by going to Administration -> System -> Troubleshooting and support tools. If all the steps were followed, everything should be green. If not, the items not in a passing state will link to documents to help you fix it.

And finally, we want to remove the firewall exception we setup to initially access JIRA during setup.

 firewall-cmd --zone=public --remove-port=8080/tcp

And now, we are done!

Congratulations! You know have a production ready JIRA instance. It’s a good bit of work, but we’ve made it! Go, take a break, relax for a few minutes. There are still steps you will need to do like add users (or preferably, a user directory), setup projects, and in general be *that* guy at your company. But for the moment, this task at least is done.

It’s been quiet the journey. I get it, this isn’t a trivial process. To help, I’ve long had a checklist that I follow to make sure I hit each step – and I’ve decided to make it public. Hey, if it’s good enough for a pilot preparing for takeoff, why not use it as an Admin preparing for a Production instance. You can access it in the link below

And Some closing notes

I’ve seen some people reminding me that the Atlassian JIRA Cloud offering is still a thing, and requires a lot less setup. And you are right! There are customers who need will need something quick, and have no need to learn how to set it up the way I have. For these customers, absolutely use JIRA Cloud. It does have it’s limit, but for small shops, it’s really the best option.

That being said, there are companies that don’t have that option. Either internal policies or relevant laws say they cannot have a cloud instance of JIRA. And they may not know the first thing about setting up a production ready instance. Those are the people I wrote this guide for.

Knowledge isn’t meant to be trapped and kept forever. It’s for us to share after we learn it, so that it can live on well beyond our time. I consider this just the start of giving back the knowledge I have gained from the greater Atlassian community.

So – as of this writing, I’m not actually sure what next weeks post will be about. I have several ideas, just not sure which one to chose. If you have anything you would like to hear me talk about, please let me know in the comments below! Until then, this is Rodney, asking “Have you updated your JIRA issues today?”