Following the Breadcrumbs: Decoding JIRA’s Logs

Well, that’s a thing. For those that don’t know, Rachel pretty much wrote the book on being a JIRA Admin. So, I guess it’s time to put up or shut up. JIRA Logging, lets do this.

For those of us in the trenches, logs are life savers. No, really. When something goes wrong within an application, logs often are our first, and sometimes only view into why it went wrong. You can tell a lot about what JIRA is doing from it’s logs – If you know what to look for.

As always, we start any look into a new aspect of JIRA by looking at the documentation. Here, we find the relevant document under Logging and Profiling:

Where to find your logs.

Finding your logs is actually pretty simple, assuming you know where your home directory is. Your log files for JIRA can be found at <jira_home_directory>/log/.

Navigating here, you will see several log files, usually along the lines of “atlassian-jira-<name>.log. If that log has had a roll-over, you will see some repeats of them, with older ones being followed by a number.

The one we will be interested in nine times out of ten is the atlassian-jira.log file.

But what about the Catalina.out file?

You my have noticed another log file in the JIRA install directory called catalina.out. This is tomcat’s log file, and while it replicates most of what is in atlassian-jira.log, it won’t have everything JIRA’s doing in there. Furthermore, it will muddy the water with additional information from Tomcat. If you haven’t changed any settings in the JIRA install directory, you won’t have a lot of reasons to go into there.

However, I have found catalina.out helpful when I’ve had an issue starting JIRA because the JVM environment was misconfigured. In that case, JIRA’s logs will have nothing, but Tomcat will report the Java error. If your JIRA is up and running though, don’t worry about this log.

Typical Logs

So, here is a snapshot of the logs after doing some typical things within JIRA:

Yeah, you may have to zoom in a bit there. But for the whole, looks like organized chaos. And well, it is. And in a live production environment, these may be zooming by so fast you may not have a chance to actually absorb what’s happening. With a bit of practice you can get to a point where you can watch the logs stream by and see only the things that look out of the ordinary…but even that’s not perfect. However, lets look at one entry, when I access the login page for JIRA.

2019-10-29 08:08:44,423 http-nio-8080-exec-23 INFO anonymous 488x35978x1 xmd6sf 192.168.<redacted>,192.168.<redacted> /rest/gadget/1.0/login [c.a.jira.index.MonitoringIndexWriter] [lucene-stats] flush stats: snapshotCount=6, totalCount=63, periodSec=486768, flushIntervalMillis=81128149, indexDirectory=null, indexWriterId=com.atlassian.jira.index.MonitoringIndexWriter@19cc4430, indexDirectoryId=RAMDirectory@5d7db2f9 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@14d78abf

So, what do we have going on here. At first glance, we notice several parts.

Time Stamp and Misc Information

First, we have the time stamp. This is often the best thing to filter off of. If you know when an event you are interested in happened, you can start around that time and work your way backwards. It can narrow hundreds of thousands of lines to maybe a few hundred.

The next bit is just an identifier that tells you what part of JIRA is generating the log. for the above entry, it’s the web server. It may take some digging to figure out what is a part of what and honestly unless you are doing log aggregation, it likely isn’t going to help you a whole lot.

Logging Level

The third bit – that’s interesting. It’s the logging level. Logging levels tell the application how much of what’s going on you want recorded, and come in five levels:

You may notice I put them onto a scale. That is because as you change the logging level, you are increasing or decreasing how much of what is going on is recorded to the logs. I could do a whole post on logging levels alone, but the gist of it is if you set debug, you can almost watch all the variables change in real time in the logs. If you set it to Fatal, JIRA will only log entries when something catastrophic happens, and even then will likely leave out details leading up to the crash.

So this leads to the question: “If not enough information is a bad thing, why don’t we set everything to Debug and log all the things?”

Bad Admin! No Treat!

If you haven’t guessed, there is a trade-off. Here, it’s disk space. If you set everything to Debug before you leave on Friday, by the time you are sitting down for dinner on Saturday, you’re going to be paged for a full disk. No really, you’ll be surprised how fast text can fill up a hard drive.

Speaking in general, you can also use Logging levels to tell if something bad is happening. The Developers will assign different events to different logging levels, based on their best guess on how important it is to log. That being said, I’ve seen items that were Warn or Error that were no big deal, and I’ve seen Info level log entries that were actually concerns. The easiest example I can think of is actually in my logs right now:

2019-10-29 08:30:11,656 Caesium-1-1 INFO ServiceRunner     [c.a.crowd.directory.DbCachingRemoteDirectory] Incremental synchronisation for directory [ 10000 ] was not completed, falling back to a full synchronisation
2019-10-29 08:30:11,656 Caesium-1-1 INFO ServiceRunner     [c.a.crowd.directory.DbCachingRemoteDirectory] INCREMENTAL synchronisation for directory [ 10000 ] was not successful, attempting FULL

This is a log level INFO, but it’s saying something causing it to fail an incremental synchronization from my LDAP Directory. In my test environment, the LDAP directory is small enough this won’t be a problem, but in a full corporate LDAP directory, this can cause a significant overhead. I’ve even DDoS’ed my own JIRA instance trying to do a full directory sync. And again, this is only a “INFO” level log…

Identifier

After the Log Level, we have the identifier information. For web traffic and actions, this will tell you whose account initiated the action that resulted in the log. If you have a problem that only one user is reporting, this can help as you can search the log for that username. Of if you want to know if that person did something that caused problems, there you go too.

For web traffic, this is also accompanied by a couple of IP Addresses, usually the user’s clientside IP Address, followed by the IP address of your proxy. This can be helpful for identifying traffic originating from outside your network, or investigating access that didn’t look right.

Package

After you have your identifier, you will see a section in a bracket. This is the package that generated the log. In lay mans terms, this is a more specific part of JIRA that generated the log. This can be a specific plugin (or part of a plugin), or a particular routine within JIRA. You can see the list of packages and their associated logging levels within the System -> Logging and Profiling section of the Administrative Control Panel in JIRA.

In speaking of Logging and Profiling, you can also change the logging level per package in this section as well. These changes will persist until the next time you restart JIRA. In a future post we’ll also cover how to change them permanently. You also have access to an even deeper logging level here called “Trace” – but it is recommend you only use that if you are asked to by Atlassian Support in order to help with a specific issue. Your disks will thank you.

Message

After all these details, we get to the meat of the log – the message. This is what happened, or didn’t happen, that warrants logging. This is the “What” to the rest of the logs “Who, When, and Where”. This is where you can dig in to find whatever problem you are looking for.

And that’s what’s in your JIRA Log

And that’s pretty much it. I do have one more thing for you. For me, reading white logs on a black background can make things easy to get mixed up. To help with this, I wrote up a quick little script that will look up where you JIRA Log is, and pass it through an Awk filter so that it will colorize things based off the log level for each entry. It will also follow the log, which means it will display entries in real time as they are updated into the logs, and keep doing so until you hit “Ctrl+C”.

For reference, the colors each log level will be:

  • DEBUG: Green
  • INFO: Default
  • WARN: Yellow
  • ERROR: Red
  • FATAL: Magenta
#!/bin/bash
# Rodney Nissen
# Version 3
# Updated 10/29/2019
# Update JIRAINST to point to your JIRA Install Directory

JIRAINST="/opt/atlassian/jira"
JIRAHOME="$(sudo cat $JIRAINST/atlassian-jira/WEB-INF/classes/jira-application.properties | grep "jira.home = " | cut -d ' ' -f 3 | tr -d '\r')"
JIRALOG="$JIRAHOME/log/atlassian-jira.log"

sudo tail -f -n 50 $JIRALOG | awk '
  /DEBUG/ {print "\033[32m" $0 "\033[39m";next}
  /WARN/ {print "\033[033m" $0 "\033[39m";next}
  /ERROR/ {print "\033[31m" $0 "\033[39m";next}
  /FATAL/ {print "\033[35m" $0 "\033[39m";next}  
  1
'

I hope you will get as much use out of this tool as I have. As I stated in several parts of this post, I think this is the kind of subject that deserves a series of posts, not just one. As such, I’m thinking I’m going to do at least two more – one is an in depth dive into Logging Levels and what they actually mean, and another on how to play around with your logs. But until then, I’m Rodney, asking “Have you updated your JIRA Issues today?”

Why you should care about Enterprise Releases

So, confession time. I normally like to have a post written in advance of a release so that I have time to go back to it with fresh eyes and edit it. This week however, I haven’t been able to figure out what to post. I originally thought about doing something around dressing up your instance for Halloween, but then I remembered something very, VERY important. I’m going to be brutally honest with both myself and you for a moment…

I am VERY terrible at web design…

This is perfectly alright. When I go to brand (or rebrand) a JIRA instance, I’ll usually reach out to Marketing or the website team, coordinate with them on color palette, and try out several variations. I’ve even been known to mock up a few palettes, put together a poll, and put it to a user vote. Your users have to use this platform every day. They should have some say in how it looks.

BUT – maybe this is not a lesson I’m qualified to write up…at least not yet. But this left me a day away from post time and no article. Then, while downloading a new version of JIRA for a test I was running, I noticed this:

A new JIRA release – but not just any kind of release – but an Enterprise Version.

So, why should I care about an Enterprise Release?

To understand this, first we need a bit of a history lesson.

While at VMware, we were fortunate enough to have both Atlassian TAMs and Premiere Support from Atlassian. I cannot speak highly enough of these teams – without them and learning from them, I don’t think I’d have gotten as far as I have as an Atlassian Admin.

One of the key benefits of Premiere Support is something called a Health Check. You would be able to submit your Support Zip to the team at Atlassian, and they’d take a look at it and see if there are any changes or tuning adjustments you can make to improve performance. This quickly allowed our instances to become some of the best maintained and performing instances inside VMware.

It became our policy to submit a health check prior to any upgrade we planned to make sure we were on a good footing. Often during our health check, Premiere Support and our TAMs would advise us of what they considered “Blessed” versions. That is, versions of their software that had undergone testing by Premiere Support and were considered stable enough to recommend to large enterprise clients.

Shortly before I left VMware, this concept morphed into an Enterprise version. This version was something of Atlassian’s take on Long Term Support versions. The plan is that any bugs will be backported to this Version.

What do you mean? Isn’t a version a version?

Eh – this gets sticky if you haven’t worked in software.

Software versions are something of a code. Lets take Atlassian’s releases. They often are encoded with three numbers separated by a period. For example: 8.5.0

The first number, Eight, in this is the Major or API Version number. In most software companies, they don’t increment this unless they have made major architectural changes – in lay mans terms that is to say they pretty much rewrote major parts of the code. That is why going from v1 to v2 is a big deal for them.

For Atlassian, a change in this number means they have made changes to either the underlying JAVA API or the REST API. When this changes is when I start worrying whether plugins or integrations will break. They can break on other releases, sure, but they are more likely to break when this changes.

The second number, five, is Feature Release Number. When this changes is usually when Atlassian will add new features to their products. This can sometimes either break or render obsolete plugins, but as I said that is more likely when the API Version changes. This is typically what I mean by version.

The third and final number, zero, is the Bug fix number. This is when Atlassian will fix bugs in their products, but add no new functionality. This is where the magic in Enterprise Releases comes in. For Enterprise releases, any bug fixes Atlassian makes for future versions that also affect the Enterprise Release, Atlassian will back port to the Enterprise Release. That is why most Atlassian releases will only get to a bug fix number of 3 or 4, but Enterprise releases will get to a bug fix number of 10 or 11.

This also gets to my advice about never installing software ending in Zero. These versions have not had a bug fix release yet, which means they are statistically more likely to have bugs than other versions. Don’t get me wrong, ALL software has bugs, but at least those not ending in zero have had some fixed.

Bug fix releases are also *generally* safer upgrades than Feature Releases or API Releases. That is to say, going from 8.5.0 to 8.5.1 is *generally* safer than going from 8.4.x to 8.5.0 or 7.6.6 to 8.5.0. HOWEVER, never trust things as infallible, and always test your upgrades in a pre-production environment.

But I’ve seen Atlassian back port security bugs to non Enterprise Releases? What’s up with that?

Yes, this is true. Some bugs are so insidious, so dangerous to security that Atlassian will do the work to back port them to as many versions as are still supported. You should always pay attention to Security Advisories and plan your upgrade cycles accordingly. But that being said, for an Enterprise Release you’re also going to get other bug fixes on top of the security ones, so why not go with that?

Is that the only benefit?

Not. Even. Close. Atlassian will also do performance testing on their Enterprise Releases, and tell you the result.

This can help you in your decisions around scaling and whether an upgrade is worth it. You might even be able to use this to sell the upgrade to your user base. I have yet to meet the user who didn’t appreciate a faster JIRA.

They also prepare an REST API Change log so if you are hopping from Enterprise Release to Enterprise Release, you can prepare your integrations for the upgrade that much easier

Also, to prevent you from going page by page to collect all the changes from previous versions, they also do a full change log between Enterprise releases here.

So, yeah, Enterprise Releases. They really are a good thing.

Every upgrade is a decision. Sometimes you have to have a new feature or bug fix and cannot wait until they next Enterprise Release to get it. It happens – I’ve been there. That being said, for the long term support you get, if you are limited on upgrade cycles, you don’t have a lot to lose by going with an Enterprise Release. So it’s definitely something to consider as you are going through the release notes to figure out a target version.

So for next week I’m still not sure what to write about either. However, I did have an interesting challenge on LinkedIn from last week’s post that I’m honestly considering…

That might be interesting. I still have the VM I did that on, so it might be a thing. Until then, My name is Rodney, asking “Having you updated your JIRA issues today?”

Installing JIRA from scratch, Part 3: Reverse Proxies and Final Tweaks

So welcome back to the third (and final) post about installing JIRA from a clean linux setup. Last week we deviated a bit from this series, but there was a reason. I needed the extra time to study, and well, this is going to be a long post.

Study for what you say? I’m glad you asked! As of last Thursday, I can officially add this shiny thing to my resume:

So shiny…

I also took the opportunity to get my Scaling JIRA badge (ASB-SCJ) to extend my ACP-JA. But I did pass the test! Needless to say, this is an actual video me last week.

But with self-congratulations out of the way, I think it’s time we continue on with the series!

What were we working with again?

So, as it’s been a few weeks, lets take a look at where we are. We started with a clean CentOS install on a VM with the following specs:

  • OS: CentOS 7 Linux
  • RAM: 4 GB
  • Processors: 2
  • Root Hard drive of 30 GB
  • Extra Hard drive of 50 GB to house the database and JIRA’s home directory

On top of this, we installed MySQL 5.7 and JIRA Software 8.1.3, and had everything in a working state, but running off of port 8080.

However…Security Updates…

Fun fact, did you know that before I started with my current job, I had about three weeks of posts saved up to allow me to focus on my career for a bit? That means I had the Part 2 for this series finished and in the can about two weeks before it was published. And honestly, I had done the install and captured the pictures weeks before that. Which leads me to the reason I bring this up…

On Sept. 18, I got a nice message from Atlassian, letting me know about a security vulnerability. As the version I installed was not covered under this or any previous alert, I opted not to change my post to address it – opting instead to disclose the fact now. However, I do *highly* recommend you check back with this page regularly, and plan your upgrades and installs accordingly.

So why do we need a reverse proxy?

So, for most seasoned people, this will be obvious. But I’m going to cover it because it’s always helpful to explain the why of what we are doing. Especially when it seems like an extra step.

When Tim Berners-Lee created the original http protocol, he did not imagine a world where we’d be handling everything through websites. Honestly, he was just making a mechanism to share public research papers. Being public, he didn’t foresee the need for security.

The end goal of all this is to say, if you aren’t specifically using https (<-note the “S” at the end), every time you or someone logs into your JIRA instance, you are sending that password in clear text to the server. Sounds bad? It’s because it is.

The HTTPS protocol is how we fix this. It takes the web traffic you send, encrypts it in such a way that only the server can decrypt it, and sends it on to the server.

JIRA’s tomcat can do this encryption/decryption with some work and time – however, it’s not great at it. Combine that with the fact that it will use resources in your JVM heap that could go to JIRA, it’s generally agreed that this isn’t the best solution.

So we setup what’s called a “Reverse Proxy”. It sits in front of the web traffic, and handles the encryption/decryption much better than JIRA can. That is to say, JIRA will feel much more responsive while still staying secure.

Sounds Great, how do we get started?

First we need to ask: Which service we are using as a proxy? There are several options, with two being the ones most people choose: Nginx and Apache. Today I am going with Apache for two reasons.

The first, and likely most important for me, it’s what I know. I’ve been setting up some variation of LAMP servers for over 15 years now. It’s a system I know well by now.

Second, and likely most important for you: It’s the one I found documentation for on Atlassian’s knowledge base. When in doubt, follow the docs.

So…more reading material then?

You got it! In today’s reading corner, we have “Securing your Atlassian applications with Apache using SSL” from Atlassian. Thankfully it’s the only one we’ll need this week.

I’ll be deviating from it in a place or two (mainly in the Apache Configuration), but I will note where I do this. These changes are quality of life improvements, which makes my (and hopefully your) life easier.

SSL Certificates

If you are lucky, you will have someone at your company who can grant you access to – or otherwise get – an SSL Certificate for you. Unfortunately, you are the JIRA Administrator, so if you don’t have this resource, you will need to get one yourself.

You can get them from a variety of sources, but I recommend you stick with reputable Certificate Authorities. It would very much ruin your day if you went cheap just to find Mozilla or Google pulled that CA, causing your site to stop working across the internet.

For my home environment, I use a wildcard certificate from Comodo. No matter who you use, if they are reputable, they will have a knowledge base to help you through the process, so I won’t cover that at this time, instead assuming we are starting after you have received your certificate.

When I ordered mine last, I got it within the hour, so if you have to wait to get one, you won’t have long to wait.

So, what will I get when I finish

When you finish, you’ll likely get a zip or other compressed file containing, your certificate, several other certificates, and a key file. Save the key, it will look something like this.

Now you will notice my key file is missing. Please, for all that is good, keep your key secure at all times. With that file, anyone can decrypt the packets heading to your server.

Of what you can see here, we have three chain certificates, plus my certificate (Star_folden-nissen_com.crt). You will need to know what parts are what. Transfer all these files plus your key to your server’s filesystem, taking note where each is.

So, we’re finally going to do something on the server?

Yes! First thing first, we want to make sure Apache is installed on your system. Here we’ll use a variant on CentOS called httpd. This still very much is Apache, just with focus on only putting out the most important updates – in such a way that doesn’t break your configuration.

yum install httpd mod_ssl

If you need to install it, you’ll see the familiar confirmation. Type “y”, followed by enter to proceed.

Looking at the configuration directory, we should see the following files

cd /etc/httpd/;ls -lah

For the next section, we’ll be focusing in on the conf.d folder.

Configuring Apache

I told you when I’d be deviating from the Atlassian Documentation, and where it is. I like to separate each service a reverse proxy is supporting into it’s own conf file, as it makes everything easy to look through. While not expressly explained in the documentation, it doesn’t go against it, but I’d like to be clear about any deviations I make and why.

Moving into the conf.d folder, we’ll see the following files:

  • autoindex.conf
  • README
  • ssl.conf
  • userdir.conf
  • welcome.conf

Really, the only one we’ll care about today is ssl.conf, however we’ll want to disable the rest from apache. This can be done by renaming the extension of each from .conf to anything else. For example:

mv autoindex.conf autoindex.bck

After you run this for everything but ssl.conf, you should see this:

Now lets dive into ssl.conf. Open it up with the text editor of your choice (I prefer vim, but nano is admittedly much easier to learn). You will see a bunch of settings required for ssl. We’ll keep most of these, instead commenting out anything between <VirtualHost _default_:443> and </VirtualHost> (including these two lines).

After this, we are going to create a new file, “jira.conf”, and entire the following.

<VirtualHost *:80>
    ServerName <subdomain>.<domain>.com
    Redirect Permanent / https://<subdomain>.<domain>.com/
</VirtualHost>


<VirtualHost *:443>
    ServerName <subdomain>.<domain>.com

    ProxyRequests Off

    <Proxy *>
         Require all granted
    </Proxy>

    ProxyPass / http://<subdomain>.<domain>.com:<port>/
    ProxyPassReverse / http://<subdomain>.<domain>.com:<port>/

    SSLEngine On
    SSLCertificateFile /path/to/your/cert.pem
    SSLCertificateKeyFile /path/to/your/privkey.pem
    SSLCertificateChainFile /path/to/your/first_chain.pem
    SSLCertificateChainFile /path/to/your/second_chain.pem
    ...
    SSLCertificateChainFile /path/to/your/nth_chain.pem

</VirtualHost>

A few notes here. You need to put in as many SSLCertificateChainFile entries as you have chain files, but be sure to include them all. This example list the files in a .pem format, but in reality they can be any number of different file extensions, so don’t think you need to chain it to be a .pem specifically.

But what if I have my JIRA on a sub-directory?

This is supported, but we’ll need to make Apache aware of this fact. Assuming your sub-directory is <subdomain>.<domain>/<contextpath>, we’ll get

<VirtualHost *:80>
    ServerName <subdomain>.<domain>.com
    Redirect Permanent /<contextpath> https://<subdomain>.<domain>.com/<contextpath>
</VirtualHost>


<VirtualHost *:443>
    ServerName <subdomain>.<domain>.com

    ProxyRequests Off

    <Proxy *>
         Require all granted
    </Proxy>

    ProxyPass /<contextpath> http://<subdomain>.<domain>.com:<port>/<contextpath>
    ProxyPassReverse /<contextpath> http://<subdomain>.<domain>.com:<port>/<contextpath>

    SSLEngine On
    SSLCertificateFile /path/to/your/cert.pem
    SSLCertificateKeyFile /path/to/your/privkey.pem
    SSLCertificateChainFile /path/to/your/first_chain.pem
    SSLCertificateChainFile /path/to/your/second_chain.pem
    ...
    SSLCertificateChainFile /path/to/your/nth_chain.pem

</VirtualHost>

Please note the subdirectory here includes a leading slash, but not a trailing one. Be sure yours looks like that as well, otherwise you will have problems.

Also keep note of what you have as <contextpath> – we’ll need it again here in a bit.

So what now?

Before bringing Apache online, we’ll need to test the configuration.

apachectl configtest

Assuming everything is alright, you’ll see a “Syntax OK”. It won’t catch every problem, but it will catch some, which is better than nothing.

Next we’ll need to allow firewall traffic to ports 80 and 443:

firewall-cmd --zone=public --permanent --add-service=http
firewall-cmd --zone=public --permanent --add-service=https
firewall-cmd --reload;

And finally, we need to tell SELinux that Apache can be allowed to do an HTTP Relay.

setsebool httpd_can_network_relay on

After all of these steps are completed, we can start, enable and test our JIRA proxy

systemctl start httpd
systemctl status httpd
systemctl enable httpd

And we’re done?

If you’ve done everything correctly, browsing to your JIRA Server on port 80 should automatically redirect you to port 443 and show JIRA.

Note: if you are using a subdirectory, JIRA will not come up at all. But you should still see a secure connection. We’ll fix this in a later section.

Notice the notification bar across the bottom. This is because we need to make JIRA aware it is running through a proxy.

On the filesystem, navigate to <jira_install_dir>/conf/, and open up server.xml in your text editor of choice. You should see something like this.

Under the connections section, add the following without modifying anything else in there.

scheme="https" proxyName="<subdomain>.<domain>.com" proxyPort="443"

When done, the connector should look something like this:

Remember that context path from earlier?

If you are hosting your JIRA on a subdirectory, you will need to make JIRA aware of that here as well. Go down to the context section, and where it says path, enter “/<contextpath>”. As with earlier, note the leading slash, and do not put a trailing one. It should look something like this when you are done.

Anytime you make changes to server.xml, they will not be reflected until you restart JIRA.

systemctl restart jira;tail -f <jira_home_dir>/atlassian/application-data/jira/log/atlassian-jira.log

This should restart JIRA if you have installed it as a service, and then bring up the log file for you to follow as it boots. Anytime you make changes to configuration files, it’s always a good idea to monitor the logs during startup – just to make sure the system comes up cleanly.

SO ARE WE DONE NOW!?!?!?!?!?

Uh…no. We have two last things to do. If we go to the site now, we no longer see that notification bar. HOWEVER, if we log into our admin account, we see the following message.

Here, we simply click “Update Jira’s base URL” and it should update to the URL you are accessing it from.

While we here, we also want to run a health check by going to Administration -> System -> Troubleshooting and support tools. If all the steps were followed, everything should be green. If not, the items not in a passing state will link to documents to help you fix it.

And finally, we want to remove the firewall exception we setup to initially access JIRA during setup.

 firewall-cmd --zone=public --remove-port=8080/tcp

And now, we are done!

Congratulations! You know have a production ready JIRA instance. It’s a good bit of work, but we’ve made it! Go, take a break, relax for a few minutes. There are still steps you will need to do like add users (or preferably, a user directory), setup projects, and in general be *that* guy at your company. But for the moment, this task at least is done.

It’s been quiet the journey. I get it, this isn’t a trivial process. To help, I’ve long had a checklist that I follow to make sure I hit each step – and I’ve decided to make it public. Hey, if it’s good enough for a pilot preparing for takeoff, why not use it as an Admin preparing for a Production instance. You can access it in the link below

And Some closing notes

I’ve seen some people reminding me that the Atlassian JIRA Cloud offering is still a thing, and requires a lot less setup. And you are right! There are customers who need will need something quick, and have no need to learn how to set it up the way I have. For these customers, absolutely use JIRA Cloud. It does have it’s limit, but for small shops, it’s really the best option.

That being said, there are companies that don’t have that option. Either internal policies or relevant laws say they cannot have a cloud instance of JIRA. And they may not know the first thing about setting up a production ready instance. Those are the people I wrote this guide for.

Knowledge isn’t meant to be trapped and kept forever. It’s for us to share after we learn it, so that it can live on well beyond our time. I consider this just the start of giving back the knowledge I have gained from the greater Atlassian community.

So – as of this writing, I’m not actually sure what next weeks post will be about. I have several ideas, just not sure which one to chose. If you have anything you would like to hear me talk about, please let me know in the comments below! Until then, this is Rodney, asking “Have you updated your JIRA issues today?”

Operation JIRaspi

So, I was looking at the specs for the new Raspberry Pi 4’s when they came out in July, and something hit me. They now offer a version with 4 GB of ram. If you’ve followed me for a bit, you know this is my default sizing for a new self-contained JIRA instance. I’ve seen plenty of tutorials on how to setup MySQL and Apache, but maybe….I had an idea.

The problem previously with running JIRA on a Raspberry Pi before has always come down to memory. Even if you offload the database and proxy to another system, The Raspberry Pi’s previous record of 1 GB of memory just wasn’t enough. In the best case you’d get the heap thrown into swap space (not a place JIRA wants to be), and your service will slow to a Crawl. Worst case, the system would become so resource starved it would outright kill the JVM.

However, with the four gigabyte option now available, this might just work to run JIRA. The idea wouldn’t let me go. Considering my job loss at the end of July, I couldn’t justify the expense to actually test it. But now…

Why hello there!

No, but seriously, why?

Let me be clear, this experiment is made entirely out of impracticality. I don’t intend this to be my actual JIRA System at home. Even if I got this to run, there is no way that Atlassian would support it. This is entirely for exprimentation’s sake.

I will come again and conquer you because as a mountain you can’t grow, but as a human, I can”

Sir Edmund Hillary

I am doing this so I can learn. I want to see what JIRA does under such conditions. I want to challenge myself to find workarounds and fixes to problems as they come up. In short, I want to grow.

Challenges going into it

Given I know both the architecture of Raspian and JIRA, I can anticipate a few challenges already.

The first problem I will face is the Raspberry Pi uses an ARM based processor. However, Both Atlassian’s installer and the Java that is packaged with JIRA are both compiled against the x86_64 architecture. This means I know two things off the bat:

  1. I will have to do a tar.gz install. I’ve done one before, but it was more in the context of learning enough that I could automate upgrades with Ansible. I’ll have to rely on Atlassian’s documentation for this one.
  2. I will need to figure out what version of Java JIRA is shipping with, and install a version of OpenJDK to match it. You can force JIRA to use a different JAVA install by setting the appropriate setting in the <jira_install_dir>/bin/setenv.sh, so I’m not terribly concerned about this either.

That being said, I’d be surprised if these two are my only challenges. That being said, the only way to discover them is to go through with it, so might as well get started!

Documents List

As always, I’m including a list of documents I’ll be referencing. As always, I’ll look at the Supported Platforms document. While this experiment is most definitely not a supported platform, the document will yield clues to help me proceed.

Next I’ll be referencing the guide to doing an install from the archive. As stated, this is a process I’m not terribly familiar with, so it will definitely help.

As I’m already so far out of the Supported Platforms nest, I’m going to also use MariaDB to simplify DB installation. That being said, I intend to configure it as I would MySQL – to the extent I can. This will be another “I’ll discover the problems when I get there” deal.

Raspberry Pi 4 Specs

Looking at the relevant Specs for the Raspi4, we get:

  • Broadcom BCM2711, Quad core Cortex-A72 (ARM v8) 64-bit SoC @ 1.5GHz
  • 4GB LPDDR4-3200 SDRAM
  • Gigabit Ethernet

This is good news…mostly. A quad core at 1.5 GHz should be more than enough to run JIRA, and a 64 bit processor will allow us to run it without tweaking some setting down.

This processor and bus actually support PCIe, so we actually get a true Gigabit port on this raspberry pi, which certainly won’t hurt things.

Database Install and Setup

I’ve previously done a write-up on how to install a Database for JIRA, so I won’t bore you with that retelling. The major difference is Raspbian uses apt instead of yum, but the syntax and install method isn’t terribly different.

This part actually goes smooth. The only difference in process is that MariaDB didn’t generate a temporary root password, so I was able to run the mysql_secure_installation right away with no password. Needless to say, the password situation didn’t stay that way for long.

Preparing the Operating Theater

So I want to take a couple of steps to prepare things ahead of installing JIRA. The first thing I want to do is make sure I have OpenJDK 8 installed. I do this by running the following command:

apt-get install openjdk-8-jdk-headless

After the installer finishes, I double check it by running the following command:

java -version
Expected Output of java -version

I can see here that it did install properly. After we complete this, we’ll need to prepare a user account for JIRA to run under. You do not want your JIRA running as root under any circumstances – even in this experiment.

useradd --create-home --comment "Account for running Jira Software" --shell /bin/bash jira

JIRA Install and Setup

For the most part I follow the document I listed earlier step for step. Test seemed to work really well and JIRA came right up when I tested it. Honestly, as I stated earlier I was expecting to have to modify the JAVA_HOME variable in <jira_install_dir>/bin/setenv.sh file, but that wasn’t even needed. It’s nice to be surprised when stuff just works.

The guide isn’t clear on where to setup the install and home directories, so I stuck to the defaults:

  • Install: /opt/atlassian/jira
  • Home: /var/atlassian/applicaiton-data/jira

So, time to add the JDBC connector, start it up for good, and get to configuring.

Finally a Problem

So at this point, I continued with the web based setup. Honestly, I couldn’t believe it was going so smoothly. The database connected fine, and from the log scroll I could tell it had setup fine as well. It was after the database setup that I ran into my first problem. My web browser just went blank. I attempted a JIRA restart, and same issue. I isolated it down the following error in the logs:

java.awt.AWTError: Assistive Technology not found: org.GNOME.Accessibility.AtkWrapper

Yeah….after a bit of googling, I found it was a configuration problem with openjdk. For more details, read this forum post:

The long and short of it is I needed to go into the file

  • /etc/java-8-openjdk/accessibility.properties

And comment out the line assistive_technologies=org.GNOME.Accessibility.AtkWrapper by putting a pound sign/hash/hashtag in front of it.

After doing this and restarting JIRA (again…), everything in the logs seemed to proceed normally. After waiting for a few moments for things to proceed, I refreshed my browser and it was back to the races!

Install Complete

I was able to complete the next few steps easily, and didn’t see another problem until I went to setup a sample project.

Hello Darkness, my old friend

Looking this one up, I found this article from Atlassian:

However, this ended up being a dead end as the mentioned setting was unset. Only thing I can guess at this point is it was a weird OpenJDK 8 bug. If I’m being honest, I always prefer Oracle’s JDK to OpenJDK for this reason. For Atlassian apps, oracle’s offering just seems to run better. However, a restart of JIRA and things completed as if it never happened, so I’m at a loss. I hate it when things just fix themselves.

And it’s over!

Everything is looking good and looks like we are ready to go! Looking at the system stats, I’m not seeing any evidence of it being overloaded. In fact it looks quiet conformatble:

I do notice some slowness on JIRA startup that I don’t notice on other JIRA systems. However, I’m so far out in the weeds here that it could be caused by any number of things, honestly.

The Systems health checks also passes with flying colors, So that’s also a plus!

So….what was this even?

I’m considering this a success. Starting this project, I wasn’t entirely sure it was possible, but all told it was. This could probably support a small team of 5 or 6 for some time before it needs to be upgraded to a server of some sort.

That being said, I wholly do not recommend anyone else do this in a production system. It’s definitely a curiosity, and cool, but far from practical. My main concern here is that the main system storage is an SD card – a thing that is not known to be entirely stable. You could supplement it with some USB drives in a software RAID, but even that feels risky. If you are looking to do something like this, I’d recommend just getting a traditional desktop and re-purposing it. You’ll come out more stable in the long run. But this was definitely a fun experiment.

Next week I’ll return to the “How to install JIRA” series with the finale, where we’ll take a look at some tweaks you need to make, as well as setting up an SSL reverse proxy. So until next time, I’m Rodney, asking “Have you updated your JIRA issues today?”

Installing JIRA from scratch, Part 2: JIRA

So, last week we took a look at the process to install MySQL to a server, and set it up to work with JIRA. I’d like to keep the momentum up with the actual JIRA install. Although this will feel like a lot of steps, I can usually do a DB and JIRA install in about an hour. It’s exhausting, but possible.

This isn’t a race though. Unless I have reason to rush, I’ll still take my time, double check each step, and use a checklist to make sure I’m not missing any steps. An ounce of prevention is worth a pound of cure, after all. So, lets get started!

Planning

The first question you should have in mind is your target version. Technically, you should have considered this first as it will impact what Databases and platforms are compatible, but I had already decided that I would be targeting Version 8.1.X – mainly because I already had it downloaded!

The first place I like to look when deciding a target version for an install or an upgrade is the Atlassian JIRA release notes. This will give you all the details about what versions are released, and what new features and bug fixes were included in that release. I love how Atlassian does their release notes. One of my biggest pet peeves is when a company has “Generic bug fixes” as their release notes – I mean my job is to manage a system to tell you exactly what you fixed!

Second thing I consider is also a bit of advice I’ll give to any admins out there. As a personal policy, I never install a version that ends in “0”. If it has two zeros, the policy stands doubly so. Nothing against Atlassian, they make comparatively good software.

However, I’ve worked with software companies, and remember the mad rush that follows any software release ending in zero. There are always bugs, and a dash to fix them. It’s also, incidentally, why I put a blackout window a month before through two weeks after any software release, where no changes to JIRA or Confluence are allowed.

Directory Layout

Now that you’ve have a target in mind, comes the next step. JIRA has two main directories on the file system it cares about, the Install Directory and the Home Directory. Took me forever to understand the difference, and what does what. However, it’s a good thing to learn now, as it will save you a good bit of time in hunting down problems.

Install Directory

The Install directory is where the files that make JIRA run live. With the exception of two or three configuration files, you should not modify anything in here. And if you do, be sure to follow Atlassian’s documentation.

You will typically find this directory at /opt/atlassian/jira, and this one I suggest keeping to the default location.

Home Directory

Unlike the Install Directory, JIRA’s home directory is where everything that makes your JIRA instance unique lives. Here you will find details about how JIRA connects to the Database, any saved attachments, log files, exports, and the Index.

That is why I like to keep it on a separate drive. Considering what all is stored in the JIRA Home Directory, it will grow considerably over your instances lifetime. It is also important that you have it on the fastest hardware you can afford. The speed at which JIRA can access it’s index will have a significant impact on both re-indexing times and JIRA’s performance overall.

By default, you can find the JIRA home at ‘/var/atlassian/application-data/jira’. However, if you have installed it in another location, you can use the command below to find where it is (assuming you know where the JIRA install directory is).

cat <jira_install_dir>/atlassian-jira/WEB-INF/classes/jira-application.properties | grep "jira.home = " | cut -d ' ' -f 3 | tr -d '\r'

Documents…always with the Documents

If you haven’t realized it yet, there is a lot of reading in being an admin. This week we are going to be looking at two documents: one new one and one returning one.

Our first document is Atlassian’s guide to installing JIRA on Linux. It includes a checklist of things to check before you install, and a rundown of how to do it. For this I will be using the Installer, and installing it as a service.

You can also do an install from a tar.gz archive, but this is a more intense process, and honestly one I find more ideal for automated installations and upgrades than something for a human to do. There are people who will swear by it, and in fact sometimes there’s some bug in Linux or the installer that forces you to consider an archive install, but whenever possible I’d rather not complicate things. “Keep it stupid simple” was something my engineering professors would chant whenever possible.

Our next document is the Atlassian MySQL document from last week. Eagle-eyed Admins would have noticed that I missed a step of that doc last week. This was intentional, as you cannot complete this step until you’ve installed JIRA, so we’ll be revisiting that when the time comes.

Getting the Installer

To get the installer, I’ll go to the JIRA Download Page, the click All Versions. From there, I’ll navigate to my target version, then select “Linux 64 bit”. At this time I have the same options of downloading it to my local computer and transfering it, or using wget to download it directly to the server. Be sure if you do the latter, that the URL you copy ends with the file extension “.bin”. The link to the download should be the “Get started” button that lights up after you agree to the Atlassian Software License Agreement, if you wish to copy/past it to the server.

Once you download or transfer it to your server, you will need to modify the permission so you can run the file. You can do so with the following command:

chmod 755 ./atlassian-jira-software-*.bin

This command uses a wildcard, so it should work for all future versions of JIRA – or at least until Atlassian changes the naming convention on their releases.

Running the Installer

Now comes the bit we’ve all been waiting for. Running the installer. I’m not going to lie to you, this part is going to be dry. There’s no help for it.

Anyways, to start it, just type in the file into your command line. If you wish to run JIRA as a service, you will need to either run this as root or run this in sudo.

./atlassian-jira-software-8.1.3-x64.bin

The first question it will ask is if you really want to install JIRA here. Take a moment to confirm the JIRA package and version are correct. I intended to install JIRA Software 8.1.3, so this all checks out. Hit enter or “o” followed by enter to proceed. Press “c” then enter to cancel the installer.

The next options it gives us to to select the operation we want to proceed with. Our options are:

  1. Express Install – uses all the default options. As we want to change our home directory, this isn’t the option we want.
  2. Custom Install – This is for more advanced users, it allows you to customize where JIRA is installed, where it’s home directory is, what port it uses, etc. This will be our option today.
  3. Upgrade an Existing JIRA install – not applicable here, but this is the option people are probably more familiar with.

So, considering how we want to set this up, we’ll hit “2”, then enter

The next question is in regards to our Install Directory. We want to keep this default, so we can just hit “Enter”

After this the installer will ask for the location of our home directory. As we want to change this, I entered the location for it. The installer will create any parent folders it needs to make the final structure you put here, so I find it helps to match their default structure as much as possible. Enter your chosen directory, double check it (no really….it’s going to be a headache otherwise), and hit Enter. If you wish to keep the default, just hit “Enter” without entering anything.

Next JIRA will ask you which ports you want it to use. As JIRA does not run as root, it cannot run natively on ports 80 and 443. Unless you plan on running more than one JIRA instance on the same server, you can keep to the defaults here.

The next question will ask if you want to run JIRA as a service. What this means is two fold: first JIRA will start up as part of the server boot sequence, and second you can start/stop/restart JIRA using systemctl. So…the answer to this is “Of course!”, so you can hit enter.

The last question here before JIRA is installed is asking you to confirm all the settings you entered. Just run over them to make sure they are correct, then hit enter.

It is at this point the installer will extract the files, install everything where it is needed, and setup the service. All told this process takes about a minute. After it is done, you should see this.

We actually do not want to start JIRA yet, as we have one more step to complete before we can give it a first boot. So type “n” then enter. This will complete the installer run.

After the Installer

As stated, we need to install the MySQL JDBC connector into JIRA. Due to licensing constraints, Atlassian is not allowed to distribute this as part of their software, so this is a step we must take.

To download this, we’ll need to look at the MySQL page for this. As always, choose your download method for this and move it onto the server. I will download the compressed TAR Archive as the tools for that are pretty common on linux servers.

 wget https://dev.mysql.com/get/Downloads/Connector-J/mysql-connector-java-5.1.48.tar.gz

I will then need to decompress the file with the following command:

tar -xzvf ./mysql-connector-java-5.1.48.tar.gz

After I’ve extracted the files I’ll need to copy the given file into the JIRA Install Directory.

cp ./mysql-connector-java-5.1.48/mysql-connector-java-5.1.48-bin.jar <jira_install_dir>/lib/

After this, we can finally start JIRA on the server, and start working in a web browser to finish it’s setup.

systemctl daemon-reload
systemctl start jira
systemctl status jira

The first command tells the system controller to reload what services are available – that is to say tell it to find JIRA’s service description. The second command starts JIRA, and the last is to make sure the service is up and running correctly. I recommend we take a few seconds between 2 and 3 to let things start up.

If “systemctl status jira” comes up clean, you can run the next command to make sure it comes up on boot.

systemctl enable jira

You can also use the following command to follow JIRA as it boots up. In fact, I will usually do this anytime I need to restart the JIRA service.

tail -f <jira_home_dir>/log/atlassian-jira.log

Firewalls

If you immediately try to jump onto the server on JIRA’s port, you may notice it fails to connect. CentOS and RHEL both use firewalld. This is a fact I’ll often forget, and sit there scratching my head for five minutes before feeling like an idiot. This is an easy fix, we just need to add an exception in there to allow traffic to flow on port 8080. This will only be temporary, because as soon as we get JIRA configured at set up, we’ll work on a proxy to handle traffic so it runs on port 80/443.

To add the exception, use the following command as sudo or root:

firewall-cmd --zone=public --add-port=8080/tcp

As the Database is located on the same server as the JIRA server, we do not need to open up port 3306 for MySQL traffic. However, if you’ve set up MySQL to run on a seperate box from JIRA, you will need to open up this port. However, we should be good to go for this build.

Post Install Setup

Assuming the system has come up cleanly and you’ve configured our firewall, you should see the above screen when you use your web browser to navigate to

http://<ip_addr_of_server>:8080

You will want to select the second option marked “I’ll set it up myself”, then “next”, which will take you to the appropriate wizard.

The next screen will ask you for some details on the database. Here you will select “My Own Database”, then enter the details. The database, username, and password will have been set during the “JIRA’fying this DB” section from last week’s post.

Click “Test Connection” to be sure everything is correct before hitting “Next” to proceed. This will setup the Database. Depending on your system speeds, this will take a few minutes, so you might want to go get some water.

The next step will be some customization for your instance. You will need to put a base URL that works for now. Once we have the proxy setup, we’ll need to change it, but it will work until that time. As for the Application title, I always prefer to go with something like “Company – JIRA” or “Company – Division – JIRA” as appropriate. Just something to uniquely identify this JIRA instance within your organization.

If this is a test instance, I also like to add the word “DEV” or “TEST” at the beginning, to further emphasize this is not Production.

The next screen will ask for your license key from Atlassian. Copy/Paste it from https://my.atlassian.com and click next to proceed.

The next step will ask you for a admin username and password. A word of note here: Even if you intend to use Active Directory or Crowd to manage users, you will still need at least one local admin account. The reason why is you cannot edit a directory from an account associated with that directory, even if that account has the correct Admin privileges This means you will need a local account to modify that directory.

After you setup your account, it will ask you whether you want to set up a mail server. You will want to set this up before you move to production, as this is the main way JIRA communicates issue updates. That being said, you do not have to do it now, so we’ll take a pass on it today.

Click finish, and JIRA will take a second while it finalizes some system settings. However, we are not done yet!

That’s right, we need to finalize our account setup. Here you will choose the language appropriate for you, then click “Continue”

Next it will ask you for an avatar. This really isn’t required, and you can click next right away, but why would you say no to this friendly little ghost?

And finally JIRA will ask you if you want to setup a first project. Your options are:

  • See a project in action – This will create a JIRA project with some issues, so you can see how one looks when it is actively being used.
  • Create a new project – This will create a blank JIRA project from one of the default templates.
  • Import from another tool – This will allow you to import from another tool, like Bugzilla, Github, etc

As you can see, you have a good few options here. At the very least, you should be able to get something into a CSV format. I’ve done plenty of these, and can say that it will work fairly well so long as you have good clean data.

And that’s it for now!

We’ve gone from a blank server to a functioning JIRA instance! While this is definitely a milestone to celebrate, we are not quite done yet. Next week we’ll see what settings we’ll need to modify to get web traffic going through SSL.

This will require us to modify system files, setup Apache, and change some settings with JIRA itself. That being said, if you’ve made it this far you shouldn’t have any problems moving forward.

One of the things I intend to release once I finish this series is a checklist you can use as you proceed to make sure you aren’t missing any steps. When I’m going an upgrade or other maintenance, I’ll always take time beforehand to prepare one, and print it out so I can check things off as I’m doing it. I’ll even include the pre-flight checklist to make sure we are good to go.

Next week we are taking a break from this project. The last two week’s post has inspired me, and I will take a look next week to see if you can run JIRA smoothly on one of the new Raspberry Pi 4’s. Just experimenting for experiment’s sake. After that we’ll return to this project with setting up the Proxy through Apache. So until then, I’m Rodney, asking “Have you updated your JIRA issues today?”