Category: Jira Data Center

So, when we left off as week, we had all the support system ready to go. As a review, we have the NFS Server, database, and Load balancer – each needed for JIRA Data Center to work to it’s fullest potential. Now we need to convert the actual JIRA instance into our first node for JIRA Data Center.

First, I want to address the elephant in the room. You may have noticed that this blog post is being released a little later than normal. That is because Atlassian released another vulnerability report today. This time its for Confluence Server and Data Center. I won’t be covering it today – I’d much rather get back to the JIRA Data Center install. But I’ll post it here for your own research

• Confluence Server and Data Center – Atlassian Companion Man-in-the-Middle – CVE-2019-15006

Also, as a cleanup note from last week, I forgot to mention one thing. All the systems you use should be configured to use the same NTP server for time synchronization. That is to say not the same *pool*, but the same *server*. What, me? Make that mistake? And only find it while writing this article? Never! </sarcasm>

Okay yeah, I did that. I am not perfect, and sometimes some things do slip by. But I don’t want you to make that same mistake, so I figured I’d own up to it now and not let you fall into the same problem.

Installing the JDC License

Before we do anything further, we need to install a JDC license key into JIRA. That way once we get things done and bring it online, you won’t have any issues.

To do this, we need to find our Server ID. This can be found by going to System -> System Support, then looking for “Server ID” on that page.

After you grab this, go to https://my.atlassian.com if you are getting a trial license. And honestly, I’d recommend you get a trial license while you are doing your initial testing. No need to put money down while you are doing a proof of concept.

Click on the “New Trial License”, then on the next screen select your JIRA flavor from the drop down (Either JIRA Software or JIRA Service Desk). Then once the rest pops up, select Data Center. You will then enter your Server ID we just grabbed.

This will generate a trial license that lasts 30 days. You can renew this a number of times, but it’s not infinite, so use this time wisely.

Copy your new trial license, then in your JIRA instance, go to Application->Versions and licenses, and paste it in. Then you are done with this step, and we can begin making JIRA into a proper node.

Moving the Shared Home.

From here on out, we are going to be following the doc “Installing JIRA Data Center”

• Installing Jira Data Center

Per the guide, we are setting up our shared. First, shutdown your JIRA instance.

systemctl stop jira

Once it has stopped, navigate to your JIRA Home Directory, and copy the following folders to your shared NFS directory.

• data
• plugins
• logos
• import
• export
• caches

Once copied, be sure to change the owner back to jira so that it can work once you bring the system back up.

chown -R jira:jira /data/jira/sharedhome

And this part is done! One step closer to having JDC up and running.

Setup your cluster.properties file

Now that we have the shared home setup, we need to setup the file that tells JIRA that a) It’s a JDC Node, and b) where to find the rest of it’s files. This is the cluster.properties folder, and it’s placed in the root of the local home folder.

This file will have the following settings:

# This ID must be unique across the cluster
jira.node.id = jnode1
# The location of the shared home directory for all Jira nodes
jira.shared.home = /data/jira/sharedhome

As a best practice, I like to put the system hostnames into the node id, that way we know where to look when we are having issues, but honestly it’s arbitrary, so use whatever works. The only requirement is that each node has a unique name.

There are some more options you have in the cluster.properties file, but unless you are having problems, you shouldn’t need to use them. But, I want you to at least be aware of them.

As a last setting, if you are running on linux (like we are), go to your <jira_install>/bin/setenv.sh file, and add the following line.

ulimit -n 16384

Now we need to unblock two new ports that JIRA will use to communicate between nodes.

firewall-cmd --permanent --zone=public --add-port=40001/tcp
firewall-cmd --permanent --zone=public --add-port=40011/tcp
firwall-cmd --reload

At this point we can start JIRA as a data center instance!

systemctl start jira

If all goes well, when JIRA will come up as a single node Data Center instance.

Standing up additional nodes.

Now, we can really take advantage of the super power of JDC: Adding nodes!

Now, if we want to go easy mode, we could just clone the VM and modify the cluster.properties file. But we are JIRA Guys, we don’t do easy mode here!

From a fresh install, we need to install the same version of JIRA that is on our first node. After this install, we need to see if we are using the same userid and group id for JIRA. If not, it will cause problems with NFS. So run the following on all nodes:

id jira

If any differs, it needs to be changed to match the existing nodes. Note this also means you will need to change the file permissions that belong to that particular JIRA user on that node. I could go through the guide on how to do this, but someone else has already done a much better job than I can.

• nixCraft: How to change a USER and GROUP ID on Linux for all Owned Files

With that minor annoyance out of the way, we can copy over the local home folder. The best way I found to do this is to tarball the entire thing up and scp it to the new node.

tar -czvf jira_home.tar.gz <jira_local_home>
scp ./jira_home.tar.gz <new_node>:~/

Once on the new node, unpack the tarball and move the home folder to the appropriate place. If you used default locations for both nodes, you can probably just unpack it at the root folder and it will find it’s way to the right place. Then we just make sure the permissions weren’t broken:

chown -R jira:jira <jira_local_home>

Next we modify the cluster.properties folder, changing the jira.node.id to something unique in the cluster. As an added note, if your nodes do not resolve to DNS, you will need to add the following to this file.

ehcache.listener.hostname = "<ip_addr_of_node>"

You will also need to do this for the existing nodes as well. I also like to take the extra step of adding the file share, database, load balancer, and all nodes to the hosts file of all systems involved. That way even if you do have a DNS Failure, your cluster won’t die. Host entry take the following format:

<ip_addr>    <fqdn>  <alternate_host_name>

An ounce of prevention is worth a pound of cure, especially when IT decides to move up an upgrade of a DNS node without telling anyone. Yes, that’s a true story!

At this point, you should make sure that the system has the shared home mounted. You can do this by copying the entry for the NFS share from the first node’s /etc/fstab file and placing it into the second. Then make sure the folder structure up to the mount point is present. You can uses the -p flag on mkdir to make all required folders in one command.

mkdir -p /data/jira/sharedhome

You can now mount the share with the mount command.

mount /data/jira/sharedhome

Take a second and test that the JIRA Account can write and see all files in there. Now unblock all the correct ports you need for JDC:

firewall-cmd --permanent --zone=public --add-port=8080/tcp
firewall-cmd --permanent --zone=public --add-port=40001/tcp
firewall-cmd --permanent --zone=public --add-port=40011/tcp
firwall-cmd --reload

If everything looks good, start up JIRA as you would on any other system, and follow the logs. IF all goes well, you should be able to access the node by going directly to the IP Address for it.

At this point, check the copyright info at the bottom. You should see the new node name at the bottom after the version info:

Log in, and check the Health Check, if all is good you should see green check marks on the Cluster.

Now the only thing to do is add it to your haproxy config, restart haproxy on your load balancer, and start sending traffic to your new node!

And you made it!

It’s an involved process, I tried not to lie about that. But it’s not an Everest sized challenge. You might have to get a bit outside your comfort zone and play with some technologies that as a JIRA Admin, you haven’t played with before. But my view on it is in order for us to grow as Admins, sometimes we need to step outside our comfort zones. But at least you have a road map to help guide you on your Data Center Migration!

I’m still looking to do a Lightning round Q&A session next week, so get your questions in. Anything from how I do testing and capture images for the blog, to how do you deal with a particular challenge, to what kind of server(s) do I run, it’s all on the table.

Also have an article I’m excited to write up for the New Years – which incidentally enough, is a blog day.

So until next time, my name is Rodney, asking “Have you updated your JIRA Issues today?”

Okay, okay, you guys talked me into it. Today and next week we’ll discuss when you should start thinking about Data Center, what support systems you will need to put into place, and what the process looks like for converting your single-server instance into a multi-node JIRA Data Center instance.

It was actually an email from one of you that made me decide to cover JIRA Data Center. I love hearing from people about how much they are learning from this blog, So don’t be afraid to send me a comment, use the contact form on the blog, or send me a DM on LinkedIn.

Now, some forewarning here. A Data Center install is an involved process. This is something you absolutely should practice doing on a test instance. Actually no, you should practice it several times. It’s not something you want to just do on production. So, without any more delays, lets get to this.

When should you look to convert to Data Center?

So, question time. When should you look to migrate from JIRA Server to Data Center? Is this something that will even be worth the expense and time?

This is a question that I’ve actually struggled with too. I actually pushed back against a Data Center Migration for a while there, with my reasoning being that we weren’t experiencing any significant problems with performance, why take on the extra cost and effort.

It was actually a mid-day downtime event that made me reconsider. The VM host that JIRA lived one unexpectedly went down. Now the VM infrastructure wasn’t my responsibility, but the resulting corruption of the JIRA Index was. JIRA was actually down for an additional 1.5 hours because we had to rebuild the index from scratch.

So I’m going to tell you now, don’t be me. Use actual numbers and metrics to inform your decision. Atlassian recommends you look at three things. These are not a hard and fast checklist, but a way to start the conversation about whether this is right for your organization.

1. Active User Count

The first Criteria is User base. Atlassian’s own studies have shown that organizations typically start running into performance issues on JIRA Server when supporting between 500 and 1000 active users. So if your instance has a peak load of around 450 users – it might be time to bring this up.

2. Performance Degradation

The second is actual performance. If you are experiencing regular performance degradations after you’ve done every optimization you can find – it might be time to bring this up. You can only “grease the track” so much before your single node can’t support any more.

3. Downtime and Outages

The third factor to consider is how critical your instance or and how tolerant you can be of downtime and outages. If your business needs dictate that JIRA has to be up, period, you guess it. It might be time to bring this up.

Now these do provide some numbers and guidelines, but they leave a lot up to your judgement. Take your time and consider these things carefully. This is not something to shout “Leeroy Jenkins” and run into head-first.

What support systems will I need to setup?

So you’ve looked at everything and decided, “Yes, Data Center is for me.” What next? Well, that’s actually going to be the focus of rest of today’s post. In order for JIRA Data Center to work, each “JIRA Server” Node needs access to a common shared set of resources. You can actually read more about it in this week’s Document, simply titled “JIRA Data Center”

• JIRA Data Center

As you can see, the three things we’ll need are a shared Filed System, a Load Balancer, and a Shared Database. As far as Databases go, JDC supports the same selection as JIRA Server. This means we can setup the Database the same way we did for Server, only we’ll need to tweak the settings a bit to make it friendly for Network use.

For the file share, I’ll be setting up a dedicated NFS Share Server for this function. I’ll also be setting up HA Proxy for the Load Balancer. Both of these, as well as the Database, will be running on CentOS 7 systems. These are both my preferences…you could use SMB/Windows File shares if you were running in a windows environment. Or you can run Nginx as your load balancer. If you have the funds, you can even run an F5. As I’ve stressed multiple times: Follow the supported platforms sheet, but use what you or your team knows.

Assumptions

Alright, so I am assuming a few things here. First, I am going to assume that you are at least familiar with how I setup JIRA Server, based on my posts here, here, and here, and that you have set up your Server instance following those instructions. I am also assuming that the database is currently on the JIRA Server.

These are assumptions I need to make in order to write this guide, as I need to know where you are starting from. However, I also think that you are smart enough that where your configuration is different, you can figure it out from what I’ve provided.

Setting up an NFS Share for the Shared Home

So…this is annoying. There doesn’t appear to be a good guide from Atlassian in setting this up. But that’s okay, I’ve setup NFS before for other purposes, so we got this.

Start with a fresh CentOS 7 machine. Our first job will be to install the package nfs-utils.

yum install nfs-utils -y

After this we’ll make a directory where the share will live. For monitoring ease, I’m also going to put this on it’s own drive separate from the root filesystem, but one step at a time.

mkdir /var/jdc-share

Now that we have a directory for the share, lets map it to a drive. To do this in a sustainable way, we need to find the UUID of the new drive – which for my example is /dev/sdb1

blkid /dev/sdb1

We then take this, and using the text editor of your choice, modify /etc/fstab, adding the following line:

UUID=e64ff994-a046-407e-96fe-3bc8ba149254 /var/jdc-share xfs defaults 0 0

If you have your fstab correct, all you should have to do is mount the folder. I will also check df -h to be sure it mounted as I expected:

mount /var/jdc-share
df -h

Next we need to make sure the permissions are correct so that we don’t have any issues when we configure it for NFS. To do this, we need to set the folder’s permissions and file ownership:

chmod -R 755 /var/jdc-share
chown nfsnobody:nfsnobody /var/jdc-share

Now that we have the filesystem prepared, we can configure the NFS service to actually share that folder. Open up the /etc/exports folder in the text editor of your choice and add the following line

/var/jdc-share <ip range of JIRA nodes>(rw,sync,no_root_squash,no_all_squash)

For the IP range – you might need a network engineer to help you. Most of the time though, you can get a close approximation by doing the following. Take the IP address of your first node or JIRA Server, as appropriate, and replace the last octect (number) with 0, then add a /24 to the end. So if your JIRA Server’s IP is 172.16.1.63, your IP range will likely be 172.16.1.0/24. This only works if all your JIRA nodes will be in the 172.16.1.0 subnet! Talk to your Network Engineers or whoever provisions your IP addresses to confirm!

So with that configured, start the following services:

systemctl start rpcbind
systemctl start nfs-server
systemctl start nfs-lock
systemctl start nfs-idmap

This will start your services. Now we need to make sure the Firewall won’t block your incoming nfs requests:

firewall-cmd --permanent --zone=public --add-service=nfs
firewall-cmd --permanent --zone=public --add-service=mountd
firewall-cmd --permanent --zone=public --add-service=rpc-bind
firewall-cmd --reload

Now it’s time to test. Remember, you are not done until you’ve confirmed it’s working yourself. Go to your JIRA Server and install nfs-utils, the same as we did for the NFS Server. Then to a temporary mount using the following command:

mount -t nfs <ip address of nfs Server>:/var/jdc-share /mnt

You can use a domain name for this, but DNS then become yet another point of failure, so for the support systems, I like to use IP addresses where possible. If all goes well, you should see no error. Try writing a file or two to /mnt/ from the JIRA Server, and see if you can see it on the NFS Server.

If all looks good, we can go ahead and enable those services for the NFS Server, and add a fstab entry to all your future JIRA nodes, then mount the share.

On NFS Server:
systemctl enable rpcbind
systemctl enable nfs-server
systemctl enable nfs-lock
systemctl enable nfs-idmap

This is the following line that needs to be added to the /etc/fstab in every JIRA node in your JDC deployment:

<ip address of NFS Server>:/var/jdc-share /data/jira/sharedhome nfs defaults 0 0

Make sure the mountpoint ‘/data/jira/sharedhome’ is created on each node, then manually mount the share to that node. By adding it to the /etc/fstab, it will also auto-mount on each boot.

umount /mnt
mkdir -p /data/jira/sharedhome
mount /data/jira/sharedhome

And that’s the Share. We can’t really use it until we go to transform our JIRA Server into JIRA Datacenter – unlike the Load Balancer and Database. But we’ve at least tested it working as expected.

Setting up HAProxy for use with JIRA DC

Unlike with the file share, Atlassian has some documentation around setting up your Load Balancer.

• JIRA Data Center Load Balancer Examples

So, for this we’ll also be starting with a fresh CentOS 7 instance. To install HAProxy, we’ll run the following command.

yum install haproxy -y

This will install the application on your system. To configure it, first take a backup of the file /etc/haproxy/haproxy.cfg, then open it in the text editor of your choice.

cp /etc/haproxy/haproxy.cfg /etc/haproxy/haproxy.cfg.example
nano /etc/haproxy/haproxy.cfg

Once open, remove the following sections from the default config:

• frontend main
• backend static
• backend app

After this, add the following to the bottom:

#Begin JIRA Configuration
frontend ft_web
  bind 0.0.0.0:8080
  default_backend bk_web
  
backend bk_web
  balance roundrobin
  cookie JSESSIONID prefix nocache
  server s1 <JIRA Server IP>:<JIRA Server Port> check cookie s1
#End JIRA Configuration

Once you enter this, start haproxy with the following command:

systemctl start haproxy

You should be able to go to port 8080 on the Load balancer server and get to your JIRA instance. This assumes that port 8080 is open on your JIRA server and the load balancer. If it passes, enable the service so that it survives a system restart:

systemctl enable haproxy

A note here: My configuration assumes you are going to put a proxy in front of the load balancer to handle SSL translation. You would do this similar to how we did it for JIRA Server, just pointing to the load balancer instead of the JIRA application. You can also set up HAProxy to handle SSL directly as well, but lets keep thing easy here.

Database?

The database is the easiest part of all this. If you’ve already setup the database as a remote resource, congrats, you are already done with this section. However, if you haven’t, please read on.

From yet another fresh CentOS install, follow the database setup we went through with JIRA Server. All the settings will be the same for this, save for a few details. First, we need to tweak the SQL statement that grants access to the JIRA User.

By default it looks like:

GRANT SELECT,INSERT,UPDATE,DELETE,CREATE,DROP,REFERENCES,ALTER,INDEX on <JIRADB>.* TO '<USERNAME>'@'<JIRA_SERVER_HOSTNAME>' IDENTIFIED BY '<PASSWORD>';

However, we need to make sure JIRA can log in remotely, so instead of “<jira_server_hostname>”, we’ll be putting ‘%’, so that it now looks like:

GRANT SELECT,INSERT,UPDATE,DELETE,CREATE,DROP,REFERENCES,ALTER,INDEX on <JIRADB>.* TO '<USERNAME>'@'%' IDENTIFIED BY '<PASSWORD>';

Granted, you could add a grant for every JIRA hostname, but in reality this adds alot of overhead for only marginal security gains, so it’s not really necessary.

Next we need to add a firewall rule to allow traffic to mysql:

firewall-cmd --permanent --zone=public --add-service=mysql
firewall-cmd --reload

And that will be your Database ready to host a JIRA database. Now we just need to migrate a JIRA DB to it. First thing you should do is shut down your JIRA instance as to guarentee there will be no changes to the data while you work. If you setup your JIRA instance as a service, enter the following.

systemctl stop jira

Now go to your current JIRA host, and enter the following command, using the JIRA DB’s username and password for that host:

mysqldump -h localhost --user=<JIRA_DB_USERNAME> --password=<JIRA_DB_PASSWORD> --single-transaction --quick --opt <JIRA_DB> | gzip > "jiradb-$(date +%Y-%m-%d).gz"

Transfer the resulting file, which should be named “jiradb-<today’s date>.gz”, to your new centralized DB server. Once it’s there, import it into that database using:

gunzip < jiradb-<today's date>.gz | mysql -u root -p <JIRA_DB>

You should take a moment here to connect to the new database with a remote tool, like mysql workbench, using the JIRA DB username and password. Make sure you can see your JIRA Database as you’d expect it before we can move on. If you can connect to it, JIRA should be able to connect to it, assuming no network obstructions.

Once you completed that, you should be ready to cut JIRA over to the new database. Go to the JIRA home folder on your JIRA Node. There you will find a file called, “dbconfig.xml”. Take a backup of it, then open that up in your text editor of choice.

In here we are looking for the fields “username”, “password”, and URL. Change your username and password to match your new database system.

Now startup JIRA, and monitor the logs to make sure it connects cleanly. Also, once it’s done loading, go into the UI to make sure everything looks normal. Also check under the System -> Troubleshooting and Support tools that there are no errors, and check the System -> System Info to make sure it’s connected as you expected.

If everything looks good, congratulations! Your system is now ready to be converted to JIRA Data Center – which we’ll focus on next week.

A quick note here: This only works if you are staying on the same database platform. If you are using this opportunity to migrate to another Database platform, like moving from MySQL to PostgreSQL, I suggest you read the following Documentation. However, the gist of it is, you will need to run an export of your entire instance, stand up a new instance on the new Database Platform, then Import your backup into that new instance, followed by copying over the attachments.

• Switching Databases

Can’t I have one system do all three support roles?

In an ideal world, yes. Save the resouces, run only one VM. We don’t live in an ideal world. The idea here is to spread the risk around to multiple systems so that any one point is less likely to be a problem. Having one machine perform multiple roles increases the complexity of that system, and therefore increases the likelihood of something going wrong. As my engineering professors used to say: “Keep it simple”

But you have some pretty big single points of failure you got right there.

This is a valid criticism of my configuration here. With enough time and resources, ideally you’d want to make every system here redundant – and they all support redundancy. However, is it always worth it? My goal is to point you in the right direction. In my lab setup, I have one VM server, with limited resources. There is also time to consider. I have a self-imposed deadline for these articles as well.

But if you have the time and resources, you should definitely research how to run each of these services redundantly. If our goal is no downtime, it will do a lot to guarantee that. Understand this is an example project, and not a production system.

So what’s Next?

Well, next week we’ll talk about converting your JIRA Server system into a JIRA Data Center Node, and then what you need to do to setup each additional JDC Node after that.

Also a note about the coming weeks. We are about to enter the holiday season here in the United States, so I’ll actually be off work (though on call) for the blog posts on the 25th and the 1st. Given that, I’d like to do something special for those. So, send me your quick JIRA questions and I’ll do a lightning round Q&A, assuming I get enough questions in. And remember, I am always willing to take on reader requests for topics, so even if you don’t think your question is small enough for a lightning round, ask it anyways! This very post started as a reader request!

So until then, this is Rodney, asking “Have you updated your JIRA Issues today?”