Redmine shows "create session key: disk quota exceeded"
We noticed that the Docker container for Redmine was not working anymore:
$ redmine@redmine01.typo3server.ch in ~ on git:cleanup x disk:47% [16:54:03]
$ docker-compose exec redmine bash
OCI runtime exec failed: exec failed: container_linux.go:367: starting container process caused: join session keyring: create session key: disk quota exceeded: unknown
However, the disk is not full.
Additionally, the process list shows a number of git
processes in defunct
state:
$ ps aux | grep git
690823 3823 0.0 0.0 0 0 ? Zs 17:45 0:00 [git] <defunct>
690823 4951 0.0 0.0 0 0 ? Zs 17:48 0:00 [git] <defunct>
690823 5478 0.0 0.0 0 0 ? Zs 17:49 0:00 [git] <defunct>
690823 6298 0.0 0.0 0 0 ? Zs 17:50 0:00 [git] <defunct>
690823 6743 0.0 0.0 0 0 ? Zs 17:50 0:00 [git] <defunct>
690823 7264 0.0 0.0 0 0 ? Zs 17:50 0:00 [git] <defunct>
690823 7740 0.0 0.0 0 0 ? Zs 17:51 0:00 [git] <defunct>
690823 8307 0.0 0.0 0 0 ? Zs 17:51 0:00 [git] <defunct>
690823 8428 0.0 0.0 0 0 ? Zs 17:52 0:00 [git] <defunct>
690823 8602 0.0 0.0 0 0 ? Zs 17:52 0:00 [git] <defunct>
It turns out that these are two problems. Problem number 1 can be fixed by raising system limits, see https://gkellynavarro.com/blog/2020/04/04/container-issues/
However, we don't need this if we fix problem number 2.
The defunctional git processes are created by a cronjob which runs every 5 minutes on this server:
$ crontab -l
SHELL=/usr/local/vzscripts/sfoutputtosyslog
# +------------------------------------ minute (0 - 59)
# | +---------------------------- hour (0 - 23)
# | | +-------------------- day of month (1 - 31)
# | | | +------------ month (1 - 12)
# | | | | +---- day of week (0 - 6) (Sunday=0 or 7)
# | | | | |
# 10 2 * * * <command>
*/5 * * * * cd /home/redmine && docker-compose exec -T --user=redmine redmine bash -c "cd /var/git/repositories/Packages/TYPO3.CMS.git && git fetch origin +refs/heads/*:refs/heads/* && git reset --soft && cd /usr/src/redmine && bundle exec rake redmine:fetch_changesets RAILS_ENV=production" >>/home/redmine/log/cron_fetch_changesets.log 2>&1
When running only the first part of that command (cd /home/redmine && docker-compose exec -T --user=redmine redmine bash -c "cd /var/git/repositories/Packages/TYPO3.CMS.git && git fetch origin +refs/heads/*:refs/heads/* && git reset --soft
) then the following message appears:
Auto packing the repository in background for optimum performance.
It seems like this process is started in the background, and it is killed as soon as the shell quits after running all following commands.
The solution was found here: https://stackoverflow.com/questions/8633981/what-does-auto-packing-the-repository-for-optimum-performance-mean
We can instruct Git to clean up the repository in the foreground by setting git config gc.autoDetach false
:
$ docker-compose exec --user=redmine redmine bash
redmine@28cf14a10e0b:/usr/src/redmine$ cd /var/git/repositories/Packages/TYPO3.CMS.git
redmine@28cf14a10e0b:/var/git/repositories/Packages/TYPO3.CMS.git$ git config -l
core.repositoryformatversion=0
core.filemode=true
core.bare=true
remote.origin.url=https://git.typo3.org/Packages/TYPO3.CMS
redmine@28cf14a10e0b:/var/git/repositories/Packages/TYPO3.CMS.git$
redmine@28cf14a10e0b:/var/git/repositories/Packages/TYPO3.CMS.git$ git config gc.autoDetach false
redmine@28cf14a10e0b:/var/git/repositories/Packages/TYPO3.CMS.git$ git config -l
core.repositoryformatversion=0
core.filemode=true
core.bare=true
remote.origin.url=https://git.typo3.org/Packages/TYPO3.CMS
gc.autodetach=false
This will immediately fix the problem. After a reboot, there are no more defunct processes, and there should be (so we think) no more errors about disk being full.