05 - Docker Workshop
Creating a dockerfile
Adding python
FROM python:3.7-alpineDifferent Linux distributions (here for Python images):
Stretch (Debian distribution Version 9.8)
Slim-Stretched (based on Debian:Stretch, uses a light desktop environment -> suitable when remote access is not required)
Alpine (smallest space amount needed -> therefor especially used in containers)
Windows Server Core -> WON’T COVER
OnBuild -> WON’T COVER
Python’s dockerfile specifies the installation of an Operating System for the image to run on:
FROM alpine:3.9
Adding Alpine Linux in Version 3.9 (Alpine’s dockerfile itself states ‘FROM scratch’, which means, that no OS is installed here)
Create a folder for the flask application (RUN executes any shell command)
RUN mkdir /appSpecify a working directory (all commands from here use it as the relative root dir)
WORKDIR /appCopy files to application directory (COPY <src> <dest>)
COPY requirements.txt requirements.txtor using the absolute path (not needed as working directory has been set before):
COPY requirements.txt /app/requirements.txt
You cannot copy any files or directories that are above the working directory.
Install Python packages from requirements.txt
RUN pip install -r requirements.txt
packages installation should be done before the deployment of the app (next step) as when any change to the application code occurs, all packages will be installed again, even though there are no changes
this happens since all steps are repeated after the first step with a change is detected
Copy all application files to working directory (using relative ‘.’)
COPY . .
When the source code has changed all steps after this step are executed -> that’s why the dependencies installation steps should be put before this one.
Add metadata to your docker image (optional, but useful)
LABEL maintainer="Arne Wohletz <arnewohletz@gmx.de>"You should try to keep the amount of layers in your docker images as low as possible
Each additional layer increases filesize and built time of your docker image
Each label creates a new layer, so it is wise to chain multiple labels together to form a single layer
The image (FROM) used in one image can be cached to be used in other images, e.g.
FROM python:2.7-alpine
can be used in multiple images without saving a copy for each image on the Docker Host system (decreases file size).
Here, two metadata key-value pairs are added to the same label and layer:
LABEL maintainer="Arne Wohletz <arnewohletz@gmx.de>" \ version="1.0"Indentation isn’t required, but looks better.
Define the application start command
CMD flask run --host=0.0.0.0 --port=5000The CMD command is run when the application is run (whereas the RUN command is run, when the application is built)
This command defines what it supposed to be executed, when the image is executed
The CMD command can be overridden from users via the command line (if they want to)
The CMD command should always be the last entry in the dockerfile
Building a docker image
After the dockerfile is complete, run this command from inside the projects working directory.
docker image build -t web1 .-> builds the docker image with the tag ‘web1’ of the current directory (‘.’)
-t defines a tag name
The build process creates different layers for
Running the command again without any changes to the dockerfile lets the image build finish almost instantly
Inspect the docker image
docker image inspect web1
see the output:
version is set to latest, since no version was supplied during build
Cmd shows the exact command being executed on the image
Created layers are displayed at the bottom
Specify a version
docker image build -t web1:1.0 .
List all docker image on the host system
docker image ls
Delete a docker image
docker image rm web1:1.0
Delete via image id
docker image rm -f <image_id>
You may only use the first few digits of the id to delete all tags referencing an image with the same id (as done when adding additional tags to an existing image)
-f option enforces the removal
Push docker image to docker hub (and pull it again)
First you need to authenticate your Docker CLI to Docker Hub in order to push images to it
docker login
-> Enter your username and password
Hint
This step does not required to be repeated as your authentication data is written to a config file located at ~/.docker/config.json (OSX, Linux) or %USERPROFILE%/.docker/config.json (Windows)
Tag your image with your Docker Hub username -> image copy with new name on host system is created (uses same image id as original)
docker image tag web1 <dockerhub_username>/web1:latest
Push the image to docker hub
docker image push web1 <dockerhub_username>/web1:latest
You can find the image under your account / Repositories
When deleting the image from the host sytem (see last chapter), the image from the docker hub repo can be pulled via
docker pull web1 <dockerhub_username>/web1:latest
Running docker containers
Get all currently running containers
docker container ls
Get all containers including stopped ones
docker container ls -a
Remove a (stopped) container
docker container rm <four_first_digits_of_container_id>
Start your app image inside a container (here: web1)
docker container run -it -p 5000:5000 -e FLASK_APP=app.py web1
-it sets the container up to be interactive:
allows for interrupting the app via ctrl+c
provide real-time input (e.g. navigating the filesystem)
-p specifies ports used by docker container an the application:
-p <bind_port_of_docker_host>:<bind_port_of_docker_container> (here the application uses
5000 internally and is supposed to be accessible via 5000: CMD flask run –host=0.0.0.0 –port=5000)
-e sets an environmental variable. Flask expects the FLASK_APP variable being set to the file that launches the flask server (so contains the flask app), which is in app.py
Hint
Get overview of all containers (including stopped containers)
docker container ls -a
Start it with a given name (instead of letting docker choose one) + remove it once it is stopped (otherwise stopped containers will stay):
docker container run -it --rm --name web1 -p 5000:5000 -e FLASK_APP=app.py web1
–rm automatically removes container, after it is killed
–name adds a name to the container (if none is given, a random one is assigned to it -> name must be unique)
Run a container in detached mode (in the background) -> the only output is the container id:
docker container run -it --rm --name web1 -p 5000:5000 -e FLASK_APP=app.py -d web1
-d launches app in detached mode, which means the container is run in the background
To check logs from running or closed containers run
docker container logs <container_name>
Container name can be defined via -d)
Prints out all log content
To print current logs to the command line type
docker container logs -f <container_name>
-f stands for foreground
Checking the current resource usage of running docker containers
docker container stats
showing disk usage, cpu usage, memory usage, network i/o
Running multiple instances of the same container in parallel requires
unique name
unused internal host port
so
docker container run -it --name othername -p 5000 -e FLASK_APP=app.py -d web1
-> leaving the first port out (host port) assigns a random port on the docker host
Automatically restart a docker container, if it crashed or docker daemon is restarted
docker container run -it —name another name -p 5000 -e FLASK_APP=app.py -d -—restart-on-failure web1
–restart on-failure enables the automatic restart of the container when it crashes (cannot be combined with –rm). You can test it by running ‘sudo service docker restart’ on Linux and see if the container is restarted once Docker is back. If stopped manually, it does not restart.
Stop a running container (command takes a few seconds)
docker container stop
Making Code Changes with Volumes
Activate the debugger in flask
docker container run -it --name withDebug -p 5000:5000 --rm -e FLASK_APP=app.py -e FLASK_DEBUG=1 web1
Make some changes in the code (app.py) -> restarting the container won’t apply the code changes
Manual approach
The manual approach is to rebuilt the image
docker image build -t web1 .Starting the new container applies the code changes
Approach with volumes
Mounting the application code into the running container
Starting the container with the volume flag (-v)
docker container run -it --name withDebug -p 5000:5000 --rm -e FLASK_APP=app.py -e FLASK_DEBUG=1 -v $PWD:/app web1
-> Multiple volumes flags can be supplied
You can inspect the mounted volumes by running (providingthe container name)
docker container inspect web1
and checking the “Mounts” section content
Important
Changes applied within the mapped source directory are applied to the docker container’s mounted copy
CAREFUL: Changes made to the container’s mounted volume are also made to the source directory
Changes made outside a mapped directory are not applied to running containers (out of scope)
Debugging Tips and Tricks
Hint
Problem
Cannot interact with alpine filesystem of a running container using VirtualBox (Docker Toolbox, Windows issue only)
Cause
inotify (notifies OS about file system changes) does behave differently on alpine. It seems no problem when alpine is used on a Linux docker installation, though.
Solution
Change alpine to slim in your dockerfile: python3.7-alpine -> python3.7-slim-stretch
Connecting to a running container’s file system
docker container exec -it <container_name> sh
Command is useful to make sure, you are really running what you think you’re running
Exit the exec session by hitting Ctrl+D
You can use any other command instead of ‘sh’ e.g. flask –version which prints the flask version info to the local terminal window
Important
All commands docker executes on the containers host OS are executed as root user. If new files are created by docker, they are owned by root (Linux only).
To prevent this use the –user flag
docker container exec -it --user "$(id -u):$(id -g)" web1 touch hi.txt
Linking Containers with Docker Networks
Install redis container
docker pull redis:5.0-alpine
Docker containers can run on multiple networks
Same as computers in general, docker containers must be on the same network
If they are scattered across multiple networks, a bridge must be created between the various docker container networks
Docker creates several networks on it own. Check them out
docker network ls
bridge: stands for the outside docker network (listed in your networks when running ifconfig)
‘host’ and ‘none’ are required by the docker daemon
if no other network is specified for a container, the bridge is used as default
Inspect bridge network
docker network inspect bridge
-> “Containers” entry lists all running containers on that network
Running containers on the same network can talk to each other via their local IP addresses (here web2 and redis are launched to the bridge network)
Check out a running containers IP address (here: redis)
docker exec redis ifconfig
Check their communication by piing the other container
docker exec web2 ping 172.17.0.2
Check the hosts mapping file -> other container is added with ip and its container ID
docker exec redis cat /etc/hosts
Creating a new network for Docker is useful, as it automatically the containers names to its current IP addresses, even if those change, due to their unique ID (automatic DNS)
docker network create --driver bridge <some_network_name>
Launch a container on a specified network (created a network named ‘firstnetwork’)
docker container run -itd --name web2 -p 5000:5000 --rm -e FLASK_APP=app.py -e FLASK_DEBUG=1 --net firstnetwork -v $PWD:/app web1
Now we can access each container on firstnetwork with their name (not their IP addresses anymore)
docker exec web2 ping redis
Persisting data to your docker host
Restoring data after the container has been stopped
For storing data of redis we will use a named volume (web2_redis:/data)
Perfect for databases
Create a named volume
docker volume create web2_redis
List all volumes of docker host
docker volume ls
more details
docker volume inspect web2_redis
Important
The mount point is not accessible on the local system (at least on OSX it is not) -> this is a design concept, the path is a place on the “remote” machine filesystem (from: https://github.com/docker/for-win/issues/6875)
Add the data volume to redis container when starting it
docker container run -itd --name redis -p 6379:6379 --rm -e --net firstnetwork-v web2_redis:/data redis:5.0-alpine
Hint
Redis specific stuff: Save current data
Since redis is a in-memory storage, it must be manually saved to the file system.
docker exec redis redis-cli save
This will create a dump.rdb file within the /data directory (on local redis container mount point) and on root dir of the redis container.
When stopping the container and restarting it, previous data is still there
This method is applicable for all kind of database -> you must merely find out, where such data needs to be stored inside the container (/data for redis)
Note
There should be no data stored inside the docker container once the application is running in production, since the application is supposed to be easily installable on various systems (portable)
Optimizing your Docker images (decrease your docker images size)
Use a .dockerignore file to reduce image size and not publish secure files:
during container creation docker will evaluate the file when COPY or ADD commands are executed
WORKDIR is the default start path for all ignore paths
.dockerignore #ignore the file itself
.git/ #ignore a directory
.foo/\* #ignore all files within a directory (but not the dir itself)
\**/*.txt #ignore all files with txt extension
!special.txt #make exception to above rule for special.txt file
-> Check out the updated dockerfile run script in the repository
Make .dockerignore a whitelist (instead of a blacklist)
*
!keepthisfile.txt
Running scripts when a container starts
Usage: Same docker images is supposed to be used for several projects, but each project has a slightly different setup routine
Usually separate dockerfiles are needed in which most commands will be the same for each application
It technically works, but all these files need to be maintained if the image changes
Scenario: Adding a database to all these projects (e.g. PostgreSQL), setting a own user login and table name for each project’s dockerfile -> very tedious
Add the entry point script to your dockerfile (located in root dir, not working dir):
COPY docker-entrypoint.sh /
Make entrypoint script executable
RUN chmod +x /docker-entrypoint.sh
Add entrypoint pointing to the copied script
ENTRYPOINT ["/docker-entrypoint.sh"]
The used shell script:
within your entrypoint script, you can print terminal message by using echo (shell) e.g.
echo "The Dockerfile ENTRYPOINT has been executed!"
an environmental variable is set (with a default value)
export WEB2_COUNTER_MSG="${WEB2_COUNTER_MSG:-some default value if not specified}
${WEB2_COUNTER_MSG} is a shell variable, which can be overridden by passing it into the docker command using the -e flag
Hint
Python is able to access environmental variables by using:
os.getenv(<env_var_name>) e.g os.getenv('WEB2_COUNTER_MSG, '')
Explaining exec “$@”
Dockerfile’s CMD instruction passes whatever comes after to a default entry point (if none is defined)
This default entry point is “/bin/sh -c” (run whatever I pass as a command (-c) in /bin/sh)
Since our entrypoint now overrides this default, the “CMD flask –host …” command actually does this
bin/sh -c "$ && flask --host=0.0.0.0 --port=5000"where $ stands for our docker-entrypoint.sh content
the last command
exec "$@"then does take every command given by you on the command line (e.g. docker container run …) and executes it ($@ stands for all command-line arguments)
-> basically continues the docker command
-> The ENTRYPOINT defines what to run when using CMD in a dockerfile
Used for
database migration
nginx config modification
Cleaning up after yourself
Get info
Check out all containers (running and stopped)
docker container ls -a
Check docker’s currently used disk space
docker system df
Check docker images
docker image ls
* images with name <none> are not used by anything -> they're dangling
* these may be failed builds or builds that were replaced by a new image using the
same tag (or both using the 'latest' tag)
* dangling images are declared as reclaimable
Get containers and images spaces usage together (verbose output)
docker system df -v
Get system info
docker system info
(also verifies that docker daemon runs well)
Cleaning up
Remove all stopped containers, unused volumes and networks and all dangling images
docker system prune
Stop multiple containers
docker container stop <containerA_name> <containerB_name> ...
Stop all running containers (at least one must be running, otherwise reports error)
docker container stop $(container ls -a -q)
-q means quiet mode, and outputs the container id as string
Removing multiple images
docker image rm <imageA_name> <imageB_name>:<tag_name> ...