Debugging a container that won't start


By Phil Helm

To implement an architecture that takes advantage of container technology, one way to persist the data has been to use a separate data container. In my case, I am using Jenkins in a container, and I want to save the configuration/jobs/history from my CI flow. Ideally, this container would a part of some kind of DR process, but in my case I simply wanted to show that the primary (jenkins process) container was disposable without the need to rebuild jobs.

During my most recent (of many) laptop restart, I noticed my jenkins data had disappeared. Actually, what I noticed was that there was a dummy “test” job that I had created as part of a previous attempt at building out Jenkins jobs. I wasnt sure why this suddenly happened, so I started debugging. I tried starting the data container (which I later realized did not need to be started in the first place) and it immediately failed and stopped. Although I can now move on to figure out why my data container is not working with my primary container, I went through some debugging steps that I thought may be of use to some.

1. Find out if the Docker daemon is running

Since I am running on Ubuntu, I built this VM to use Upstart as the boot time start mechanism. This means I need to use it’s utilities to check the status:

> sudo status docker

docker start/running, process 2841

so far so good.

2. Start your container, see if it’s running.

> sudo docker start xxx (where ‘xxx’ are there first 3 characters of the docker container)

> sudo docker ps

When I did this, nothing was listed..

CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES

jenkinsadmin@ubuntu:~/jenkins-data$

so that tells me the container didnt start correctly. Then I did a

> sudo docker ps -a

now i see my container:

CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES

6a721ae2d232 myjenkinsdata “echo ‘Data container” 4 weeks ago Exited (0) About an hour ago jenkins-data

under the STATUS column, I see it is “Exited”. Not good.

3. Check the container log

> sudo docker logs xxx

where ‘xxx’ are the first three characters of the container. This should show you any message output during container initialization. In my case, it was

jenkinsadmin@ubuntu:~$ sudo docker logs 6a7

[sudo] password for jenkinsadmin:

Data container for Jenkins

this “data container for jenkins” message corresponds to the final entry in my Dockerfile,

CMD [“echo”, “Data container for Jenkins”]

that at least told me the commands in the Dockerfile were executing.

4. Commit the container changes to a new image

The idea here is that if there is corruption in the container, you can determine whether the image itself is also at fault by committing the changes (to a new image) and creating a new container from that new image.

>docker commit –change “debug container vs image” xxxxxxxxx <repo>/<image>:<version> where ‘xxxxxxxxxxx’ is the container id.

now you can execute the “docker run” command (as you did originally to create the container you are trying to debug)

Upcoming Events
Event
Architect Core - Online October 2020 10/13/2020
VIEW EVENT
Integration - Online NOVEMBER 2020 11/24/2020
VIEW EVENT