Fixed issue with running Docker containers
In the past 24 hours, you may have encountered errors when running Docker containers on NAT Cloud servers. If you have recently experienced problems with Docker, please try your actions again now, everything should work!
It turned out that so many Docker containers were running on your servers that the hypervisor faced Linux kernel limits and could not create new keyrings, which are necessary for Docker to work.
We are sharing the solution to this problem, as there is almost no information on this topic on the internet, either no one encountered this problem, or simply did not share the solution.
The error message you might have seen:
docker: Error response from daemon: failed to create task for container: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: unable to join session keyring: unable to create session key: disk quota exceeded: unknown.
Here is a more detailed description of the incident: due to the features of LXC virtualization, some common kernel limits apply to all LXC containers set at the hypervisor level. One of these limits is the limit on the number and size of keyrings (key storage in the Linux kernel), and it is shared by all LXC containers running on the hypervisor. One day, due to the large number of saved keys, these limits were exceeded, and problems arose, including the inability to start Docker containers.
We initially thought the limit on the number of keys was exceeded and increased it, but this did not help in our situation. Eventually, we discovered that there is also a limit on the size of the storage, and by default, it is set to only a few kilobytes, which was insufficient in our conditions. The problem was solved by increasing the limits to normal values.
We set these limits with a margin and will monitor them in the future, and we share the solution with you:
sysctl -w kernel/keys/maxkeys=1000000sysctl -w kernel/keys/maxbytes=52428800
To save these settings after a reboot, you also need to create an additional sysctl configuration file with the following command:
echo "kernel.keys.maxkeys=1000000kernel.keys.maxbytes=52428800" > /etc/sysctl.d/99-docker.conf
For clarification, our NAT Cloud server clients do not need to do anything; we have already fixed everything. These settings are made at the hypervisor level. Now Docker works correctly on all our servers, and you can use it without any limitations. This instruction will be useful for other providers or users who decide to run several hundred containers on one server.
