Properly setting up Redis and Sidekiq in production on Ubuntu 16.04

Properly setting up background job processing in production involves quite a few steps. This is an up-to-date, high-level overview of the steps involved in setting up Redis and Sidekiq on the latest version of Ubuntu (16.04). Details of this configuration may change, so I won’t be explaining everything step by step and we will use official documentation wherever possible.

This article includes the following;

Installing and configuring Redis
Security basics
Installing Sidekiq
Running Sidekiq with Systemd
Deployment with Capistrano

1. Installing and configuring Redis

We start with Redis, the in-memory data structure store that Sidekiq uses to process background jobs. To install Redis, we will follow the quickstart guide from the official documentation.

the Redis quickstart guide

To install Redis you can either use apt-get or install from source. The recommended way (following the quickstart guide) is compiling it from source with the following commands (using a Linux package manager is discouraged, although not everyone agrees).

  wget http://download.redis.io/redis-stable.tar.gz
  tar xvzf redis-stable.tar.gz
  cd redis-stable
  make

This includes redis-server and redis-cli which we will be using later on in this tutorial. In the redis-stable folder, you can run the following commands to copy the executables to you local usr bin folder so you can run the redis-server and cli commands from your home / user directory.

sudo cp src/redis-server /usr/local/bin/
sudo cp src/redis-cli /usr/local/bin/

You can start and test Redis with the redis-server and redis-cli commands as explained in the quickstart guide. This is fine for testing purposes. However, this is not the recommended way to run Redis in a production environment.

The Redis quickstart guide has a section called ‘Installing Redis more properly’. This setup involves using an init script and configuration file to set up a Redis instance on a specified port. This approach is ‘strongly recommended’. The steps to properly install Redis are clearly explained in the quickstart guide and I recommend to read and follow these steps before continuing.

2. Security

Redis is not optimized for security by default. So we have to take some measures to keep our Redis instance secure. You can start with reading the ‘Securing Redis’ section in the quickstart guide. There are four recommendations listed, we will focus on the first two.

1 . Make sure the port Redis uses to listen for connections is firewalled.

The first recommendation is to set up a firewall to limit access to our server and Redis instance. We can do this by configuring UFW (Uncomplicated Firewall), ufw is installed on Ubuntu by default.

To add basic firewall security, follow the first 5 steps in the following guide (up until Specific Port Ranges). This includes adding default policies for connections, allowing ssh and only allowing http and https connections.

How to setup a firewall with ufw on Ubuntu 16.04

2 . Binding to localhost and protected mode

If you followed the steps in ‘Installing Redis properly’, this should already be configured by default. Open your Redis instance configuration file in /etc/redis/6379.conf. The bind directive is the uncommented line with bind 127.0.0.1. This will make sure you can only access Redis locally. Another security layer added by default is ‘protected mode’. You should not disable this unless your requirements include the option for other hosts to connect to Redis.

Step 3 & 4 are optional and can be added depending on your applications requirements.

3. Installing Sidekiq

Install Sidekiq by adding it to your gemfile and deploy (or using gem install Sidekiq). By default, a configuration file is not required. If you do want advanced options you can add one, see this section in the Sidekiq wiki. Later on, we can specify the path to this configuration file when we manage starting Sidekiq with Systemd.

By default Redis uses port 6379. This is also the default port Sidekiq tries to connect to. If you’re using a different port, you need to specify it in your application by using an initializer for example, see using Redis in the sidekiq wiki for more information.

4. Running Sidekiq in production using Systemd

In the official documentation, Upstart is recommended for controlling your Sidekiq processes. However, in recent versions of Ubuntu, Upstart has been replaced with Systemd. There’s has been a lot of debate in Linux land on whether this is a good thing or not, but since Ubuntu now comes with Systemd by default, that’s what we’ll be using.

Like Upstart, Systemd is an init service which handles starting of tasks and services during boot, stopping them during shutdown and supervising them while the system is running. Check out this article if you want to read more about Systemd.

To setup a Sidekiq Systemd service, we need a service configuration file. An example file can be found in the sidekiq github repo. Copy this file to your server and place it in /lib/systemd/system

There are two lines here that require adjustment to your settings;

The working directory path, change this to you application path, for example: WorkingDirectory=/home/deploy/my_app/current
The ExecStart path. This specifies the path and command to start Sidekiq. Now this could be different depending on your settings and ruby version managers (if you use one). For example, my current ExecStart path is:
ExecStart=/home/deploy/.rbenv/shims/bundle exec sidekiq -e production

You can add additional options to the ExecStart path like the location of you configuration file with and -C config/sidekiq.yml or logs with -L log/sidekiq.log. These are both optional. Sidekiq runs on default settings if you don’t specify a configuration file and the default logging is set to syslog in /var/log/syslog.

After setting up configuration, enable the Sidekiq service with:

systemctl enable sidekiq

other useful commands for controlling the service are:

systemctl {start,stop,status,restart} sidekiq

service sidekiq {start,stop,status,restart}

5. Deploying

If you use Capistrano for deployment, you can make restarting Sidekiq part of the deployment process. You can find the code example on the Sidekiq repo wiki

The only difference here is that instead of Upstart, we use Systemd. So the sidekiq:restart task should be changed to:

  task :restart do
    on roles(:app) do
      execute :sudo, :systemctl, :restart, :sidekiq
    end
  end

Executing sudo commands with Capistrano does require additional configuration on the server. The ‘deploy’ user has to be able to execute this specific sudo command without a password prompt. With the following configuration you can setup passwordless sudo for the systemctl restart Sidekiq task.

Open the sudoers file on your server with sudo visudo. On Ubuntu, this will open the file with the nano text editor. To allow passwordless sudo for our Sidekiq restart task, add this to the bottom of the file and save /exit with ctrl-x.

%deploy ALL=NOPASSWD:/bin/systemctl restart sidekiq

As always, be cautious when setting sudo privileges and using visudo. You can read more about the Sudoers file and visudo command here.

What’s next

That’s it for now! As you can see setting this up properly takes quite a few steps. Hopefully this made it somewhat more manageable. Next up is Monitoring, an important part of a robust Redis/Sidekiq setup. Coming soon!