Docker on RedHat 6.6 -- switching users problem

April 26, 2015

Tagged: docker chef eol rhel

After recent upgrade of RHEL 6.6 we found that some of our Docker containers stopped working. We found that containers lost ability to run commands as any user but root so commands like

su www-data -c 'some_command'

generated an error:

su: User not known to the underlying authentication module

Trying to run

su - a_user

would generate

Unable to cd to '/home/a_user'

running

su a_user

generated

Cannot execute /bin/bash: Permission denied
su: User not known to the underlying authentication module

The same was true about containers we built remotely or locally. The same containers would work fine on Ubuntu and even updated Centos 6.6. After several hours I tried to run

strace su www-data > strace_dump

inside of container and found that not only /bin/bash is not accessable, but any directory is closed for any user but root. After this revelation it was easy to find out that the underlying problem was drwx---- permission on / directory.

What causes this problem? I was able to reproduce it on Centos 6.6 with docker running devicemapper driver and with umask 0077:

umask 0077 docker -d &
docker pull busybox
docker run -it --rm busybox

docker run -it --rm busybox
/ # ls -la
total 56
drwx------   17 root     root          4096 Apr 26 18:20 .
drwx------   17 root     root          4096 Apr 26 18:20 ..
-rwx------    1 root     root             0 Apr 26 18:20 .dockerenv
-rwx------    1 root     root             0 Apr 26 18:20 .dockerinit
drwxrwxr-x    2 root     root          4096 May 22  2014 bin
drwxr-xr-x    5 root     root           380 Apr 26 18:20 dev
drwxr-xr-x    6 root     root          4096 Apr 26 18:20 etc
drwxrwxr-x    4 root     root          4096 May 22  2014 home
drwxrwxr-x    2 root     root          4096 May 22  2014 lib
lrwxrwxrwx    1 root     root             3 May 22  2014 lib64 -> lib
lrwxrwxrwx    1 root     root            11 May 22  2014 linuxrc -> bin/busybox
drwxrwxr-x    2 root     root          4096 Feb 27  2014 media
drwxrwxr-x    2 root     root          4096 Feb 27  2014 mnt
drwxrwxr-x    2 root     root          4096 Feb 27  2014 opt
dr-xr-xr-x  126 root     root             0 Apr 26 18:20 proc
drwx------    2 root     root          4096 Apr 26 18:20 root
lrwxrwxrwx    1 root     root             3 Feb 27  2014 run -> tmp
drwxr-xr-x    2 root     root          4096 May 22  2014 sbin
drwxr-xr-x   13 root     root             0 Apr 26 15:23 sys
drwxrwxrwt    3 root     root          4096 May 22  2014 tmp
drwxrwxr-x    6 root     root          4096 May 22  2014 usr
drwxrwxr-x    4 root     root          4096 May 22  2014 var

See a bug report about this issue.