debugging - How can I inspect the file system of a failed `docker build`?

Question

I'm trying to build a new Docker image for our development process, using cpanm to install a bunch of Perl modules as a base image for various projects.

While developing the Dockerfile, cpanm returns a failure code because some of the modules did not install cleanly.

I'm fairly sure I need to get apt to install some more things.

My question is, where can I find the /.cpanm/work directory quoted in the output, in order to inspect the logs? In the general case, how can I inspect the file system of a failed docker build command?

Morning edit After biting the bullet and running a find I discovered

/var/lib/docker/aufs/diff/3afa404e[...]/.cpanm

Is this reliable, or am I better off building a "bare" container and running stuff manually until I have all the things I need?

Answer 1

Everytime docker successfully executes a RUN command from a Dockerfile, a new layer in the image filesystem is committed. Conveniently you can use those layers ids as images to start a new container.

Take the following Dockerfile:

FROM busybox
RUN echo 'foo' > /tmp/foo.txt
RUN echo 'bar' >> /tmp/foo.txt

and build it:

$ docker build -t so-2622957 .
Sending build context to Docker daemon 47.62 kB
Step 1/3 : FROM busybox
 ---> 00f017a8c2a6
Step 2/3 : RUN echo 'foo' > /tmp/foo.txt
 ---> Running in 4dbd01ebf27f
 ---> 044e1532c690
Removing intermediate container 4dbd01ebf27f
Step 3/3 : RUN echo 'bar' >> /tmp/foo.txt
 ---> Running in 74d81cb9d2b1
 ---> 5bd8172529c1
Removing intermediate container 74d81cb9d2b1
Successfully built 5bd8172529c1

You can now start a new container from 00f017a8c2a6, 044e1532c690 and 5bd8172529c1:

$ docker run --rm 00f017a8c2a6 cat /tmp/foo.txt
cat: /tmp/foo.txt: No such file or directory

$ docker run --rm 044e1532c690 cat /tmp/foo.txt
foo

$ docker run --rm 5bd8172529c1 cat /tmp/foo.txt
foo
bar

of course you might want to start a shell to explore the filesystem and try out commands:

$ docker run --rm -it 044e1532c690 sh      
/ # ls -l /tmp
total 4
-rw-r--r--    1 root     root             4 Mar  9 19:09 foo.txt
/ # cat /tmp/foo.txt 
foo

---

When one of the Dockerfile command fails, what you need to do is to look for the id of the preceding layer and run a shell in a container created from that id:

docker run --rm -it <id_last_working_layer> bash -il

Once in the container:

• try the command that failed, and reproduce the issue
• then fix the command and test it
• finally update your Dockerfile with the fixed command

---

If you really need to experiment in the actual layer that failed instead of working from the last working layer, see Drew's answer.