Recommendation for Production Redmine install is to use Sidekiq (for Redis)

[misilot]

Redmine uses ActiveJob::QueueAdapters::AsyncAdapter as default queue adapter which is not recommended for production environment. Recommended adapter is Sidekiq which uses Redis for persistance. Configuration guide: SidekiqConfiguration.

From: https://www.redmine.org/projects/redmine/wiki/RedmineInstall#Queues-adapter

Can the example docker-compose.yml be updated to include this?

Thanks!

[petiepooo]

I too would like to see this implemented.

Might I suggest documenting an environment variable like REDMINE_QUEUE_REDIS and maybe REDMINE_QUEUE_PORT for setting the hostname and port of a Redis server, and setting the hostname results in docker-entrypoint.sh configuring Sidekiq as Redmine's queue adapter.

[yosifkit]

Since running sidekiq requires a second process it is not something we'd support running in the same container as redmine's rails server.

We'd be unlikely to add more environment variables for config since the entrypoint script is already complex and fragile. But we are open to documentation improvements and maybe even improvements to the image to allow users to run sidekiq as a second container.

[petiepooo]

I was reading up more on sidekiq and agree with your point on not running it in the same container. Having the gem be part of the base build could be useful, though, as it would potentially allow running stock images including redis and the sidekiq process in separate containers all as part of a compose stack. Right?

I know people are wanting the ability to schedule tasks and get rid of the non-prod queue warning.. if there was a documented way to do both with the right compose file, that would resolve both asks.

I appreciate the feedback.

[LaurentGoderre]

You might be able to implement a custom queue like this one:

https://github.com/aws/aws-sdk-rails/blob/main/lib/active_job/queue_adapters/amazon_sqs_adapter.rb

[petiepooo]

I'm not a good Rails dev, unfortunately. Perhaps someone who is can follow the answer at https://stackoverflow.com/questions/33563161/how-to-put-sidekiq-into-docker-in-a-rails-application better than I. An example compose file containing redmine, redis, sidekiq, and db services, plus a little extra in redmine's image (for docker-entrypoint.sh to setup sidekiq) would be ideal for my use.

[DerDummePunkt]

Since running sidekiq requires a second process it is not something we'd support running in the same container as redmine's rails server.

We'd be unlikely to add more environment variables for config since the entrypoint script is already complex and fragile. But we are open to documentation improvements and maybe even improvements to the image to allow users to run sidekiq as a second container.

I agree with the first point in principle. Docker containers should be used as process sandboxes and not VMs running many processes. Then again, there are many widely used docker images (gitlab for instance comes to mind), that are running many processes in the same container. But that is not to say that that is would you should do, I dont even want to know what gitlab had to do with gitlab-ctl to properly supervise those processes...

I'd really like to have a redmine docker image suitable for production use myself, so I'm willing to give this a shot.

Running sidekiq in another container could be "relatively" easily achieved by starting another instance of the redmine image and overwriting the command.

Aside from changes to the documentation, we at least have to change the bundle install / gemfile in the Dockerfile, as sidekiq is currently not included, unless there's something I missed here.

Although you said its unlikely you would add them, I'd also really like to add _FILE env vars for configuring redis/valkey passwords and/or usernames or redis URLs.
Imho you should treat KVSs the same as databases, dont run them without at least a password, even when only accessible from withing a docker compose network. When the feature exists for the databases, it should exist for the KVS too.

@yosifkit what do you think?

[DerDummePunkt]

@petiepooo or anyone else stumbling upon this issue,

I've thrown together what i have so far at https://github.com/DerDummePunkt/redmine_docker_compose_sidekiq,

Works quite well so far here, feel free to reach out if that gives you any trouble.

Once I have some indication from @yosifkit about possibly merging changes into this repo, I'd port those changes to the template files here and start testing on the alpine Dockerfiles as well.

[yosifkit]

Apologies for the slow reply. Looking at your entrypoint and it seems generally fine. I like the extra functionalization but am uncertain if it adds anything for functions that aren't reused (or meant to be used by others). Creating the REDIS_URL from user/pass/host/port should be left to the user. We are still very hesitant to add more environment variable configuration to the entrypoint. REDIS_URL is a value directly used by sidekiq, so it would be fine to rely on it if we add support for bundle exec sidekiq as an alternate command.

[petiepooo]

I appreciate your work on this, @DerDummePunkt, and it looks very useful. I have moved to a different job and no longer run Redmine, but the technique is valuable other places and I'm sure others can make use of what you've done.
@yosifkit, I agree that the function building the REDIS_URL could be avoided by simply setting the variable externally... except that it then holds credentials if a keyval password is included. Building that within the entrypoint allows clearer separation between secrets and configs. Given that tradeoff, I personally prefer the better secrets management that this technique allows.

[DerDummePunkt]

Apologies for the slow reply. Looking at your entrypoint and it seems generally fine. I like the extra functionalization but am uncertain if it adds anything for functions that aren't reused (or meant to be used by others). Creating the REDIS_URL from user/pass/host/port should be left to the user. We are still very hesitant to add more environment variable configuration to the entrypoint. REDIS_URL is a value directly used by sidekiq, so it would be fine to rely on it if we add support for bundle exec sidekiq as an alternate command.

My apologies in kind.

I dont quiet understand the argument against creating the env variable for REDIS_URL from the user/pass/host/port combination.
Isn't that exactly what the entrypoint already does for the database connection?

I thought that instead of re-inventing the wheel, I'd just follow that pattern.

Personally, I like using docker secrets, storing a password that way allows for pretty clear conventions when setting up a file structure for a docker compose project, the existing pattern was a good match for that.

Sure, you could also store the complete REDIS_URL in a docker secret or pass it along in any other way to the app services, but if we do that, shouldnt we then also make the same changes to the SQL Server connection?

[yosifkit]

Isn't that exactly what the entrypoint already does for the database connection?

Not really, we provide environment variables that map 1 to 1 to the respective config values in database.yaml. We don't combine them to make the connection string in the entrypoint script. Since this full URL is the value used by sidekiq and is already an environment variable, we'd just let sidekiq use it.

On a similar vein, we deprecated the custom REDMINE_SECRET_KEY_BASE (and _FILE) variables that were generating a secrets.yml in favor of the standard rails' SECRET_KEY_BASE environment variable in #365. So, if there is a standard environment variable from the beginning, we'd rather just use that instead of inventing an interface that we have to maintain.

[Silex]

Given #386 is now merged, the setup can be as simple as:

services:
  redmine:
    image: 'redmine:6.0.6'
    depends_on:
      redis:
        condition: 'service_healthy'

  sidekiq:
    image: 'redmine:6.0.6'
    command: 'bundle exec sidekiq'
    depends_on:
      redis:
        condition: 'service_healthy'

  redis:
    image: 'redis'
    healthcheck:
      test: ['CMD', 'redis-cli', 'ping']
      timeout: '5s'
      start_period: '5s'
      start_interval: '1s'

I intentionally left out the database container, the volumes and the environment variables as an exercice for the reader 😉

For both the redmine & sidekiq container, either add Gemfile.local containing gem 'sidekiq' or install the https://github.com/rammuralidhar/redmine_sidekiq plugin.

Then for configuration my preference is to add a config/additional_environment.rb which does this:

Sidekiq.configure_client do |config|
  config.redis = { url: 'redis://redis:6379/0' }
end

Sidekiq.configure_server do |config|
  config.redis = { url: 'redis://redis:6379/0' }
  config.queues = %w[mailers default]
end

But you can also use sidekiq.yml, REDIS_URL, modify the command bundle exec sidekiq with some flags for the queues... etc

[fhaefemeier]

@Silex it sounds simple but I am a little bit confused... (and have no deeper knowledge about Sidekiq, Ruby, ...)

Do I need a fully configured Redmine (with plugins, ...) or is it enough to use the base image with configuration for database, ...? Need sidekiq container a connection to the redmine database?

[Silex]

Base image is enough, but as I said you need to add files to the container so it knows about sidekiq/redis (Gemfile.local or plugins for the sidekiq gem, then config/additional_environment.rb (or REDIS_URL?) for the redis config).

You can do that through volumes, for example here's how I modify the configuration:

    volumes:
      - type: 'bind'
        source: './data/redmine/config/configuration.yml'
        target: '/usr/src/redmine/config/configuration.yml'

AFAIK Sidekiq container requires access to the db because the mailer will load info from it.

But overall you need to learn more about Sidekiq.