Chapter 1: Docker
Properly configuring a local development environment remains a steep challenge despite all the other advances in modern programming. There are simply too many variables: different computers, operating systems, versions of Django, virtual environment options, and so on. When you add in the challenge of working in a team environment where everyone needs to have the same set up the problem only magnifies.
In recent years a solution has emerged: Docker. Although only a few years old, Docker has quickly become the default choice for many developers working on production-level projects.
With Docker it’s finally possible to faithfully and dependably reproduce a production environment locally, everything from the proper Python version to installing Django and running additional services like a production-level database. This means it no longer matter if you are on a Mac, Windows, or Linux computer. Everything is running within Docker itself.
Docker also makes collaboration in teams exponentially easier. Gone are the days of sharing long, out-of-date
README files for adding a new developer to a group project. Instead with Docker you simply share two files–a
docker-compose.yml file–and the developer can have confidence that their local development environment is exactly the same as the rest of the team.
Docker is not a perfect technology. It is still relatively new, complex under-the-hood, and under active development. But the promise that it aspires to–a consistent and shareable developer environment, that can be run either locally on any computer or deployed to any server–makes it a solid choice.
In this chapter we’ll learn a little bit more about Docker itself and “Dockerize” our first Django project.
What is Docker?
Docker is a way to isolate an entire operating system via Linux containers which are a type of virtualization. Virtualization has its roots at the beginning of computer science when large, expensive mainframe computers were the norm. How could multiple programmers use the same single machine? The answer was virtualization and specifically virtual machines which are complete copies of a computer system from the operating system on up.
If you rent space on a cloud provider like Amazon Web Services (AWS) they are typically not providing you with a dedicated piece of hardware. Instead you are sharing one physical server with other clients. But because each client has their virtual machine running on the server, it appears to the client as if they have their own server.
This technology is what makes it possible to quickly add or remove servers from a cloud provider. It’s largely software behind the scenes, not actual hardware being changed.
What’s the downside to a virtual machine? Size and speed. A typical guest operating system can easily take up 700MB of size. So if one physical server supports three virtual machines, that’s at least 2.1GB of disk space taken up along with separate needs for CPU and memory resources.
Enter Docker. The key idea is that most computers rely on the same Linux operating system, so what if we virtualized from the Linux layer up instead? Wouldn’t that provide a lightweight, faster way to duplicate much of the same functionality? The answer is yes. And in recent years Linux containers have become widely popular. For most applications–especially web applications–a virtual machine provides far more resources than are needed and a container is more than sufficient.
This, fundamentally, is what Docker is: a way to implement Linux containers!
An analogy we can use here is that of homes and apartments. Virtual Machines are like homes: stand-alone buildings with their own infrastructure including plumbing and heating, as well as a kitchen, bathrooms, bedrooms, and so on. Docker containers are like apartments: they share common infrastructure like plumbing and heating, but come in various sizes that match the exact needs of an owner.
Containers vs. Virtual Environments
As a Python programmer you should already familiar with the concept of virtual environments, which are a way to isolate Python packages. Thanks to virtual environments, one computer can run multiple projects locally. For example, Project A might use Python 3.4 and Django 1.11 among other dependencies; whereas Project B uses Python 3.7 and Django 2.2. By configuring a dedicated virtual environment for each project we can manage these different software packages while not polluting our global environment.
Confusingly there are multiple popular tools right now to implement virtual environments: everything from
Pipenv, but fundamentally they all do the same thing.
The important distinction between virtual environments and Docker is that virtual environments can only isolate Python packages. They cannot isolate non-Python software like a PostgreSQL or MySQL database. And they still rely on a global, system-level installation of Python (in other words, on your computer). The virtual environment points to an existing Python installation; it does not contain Python itself.
Linux containers go a step further and isolate the entire operating system, not just the Python parts. In other words, we will install Python itself within Docker as well as install and run a production-level database.
Docker itself is a complex topic and we won’t dive that deep into it in this book, however understanding its background and key components is important. If you’d like to learn more about it, I recommend the Dive into Docker video course.
Ok, enough theory. Let’s start using Docker and Django together. The first step is to sign up for a free account on Docker Hub and then install the Docker desktop app on your local machine:
This download might take some time to download as it is a big file! Feel free to stretch your legs at this point.
Once Docker is done installing we can confirm the correct version is running by typing the command
docker --version on the command line. It should be at least version 18.
$ docker --version Docker version 19.03.2, build 6a30dfc
Docker is often used with an additional tool, Docker Compose, to help automate commands. Docker Compose is included with Mac and Windows downloads but if you are on Linux you will need to add it manually. You can do this by running the command
sudo pip install docker-compose after your Docker installation is complete.
To confirm Docker Compose is correctly installed run the command
$ docker-compose --version docker-compose version 1.24.1, build 4667896b
Docker Hello, World
Docker ships with its own “Hello, World” image that is a helpful first step to run. On the command line type
docker run hello-world. This will download an official Docker image and then run it within a container. We’ll discuss both images and containers in a moment.
$ docker run hello-world Unable to find image 'hello-world:latest' locally latest: Pulling from library/hello-world 1b930d010525: Pull complete Digest: sha256:b8ba256769a0ac28dd126d584e0a2011cd2877f3f76e093a7ae560f2a5301c00 Status: Downloaded newer image for hello-world:latest Hello from Docker! This message shows that your installation appears to be working correctly. To generate this message, Docker took the following steps: 1. The Docker client contacted the Docker daemon. 2. The Docker daemon pulled the "hello-world" image from the Docker Hub. (amd64) 3. The Docker daemon created a new container from that image which runs the executable that produces the output you are currently reading. 4. The Docker daemon streamed that output to the Docker client, which sent it to your terminal. To try something more ambitious, you can run an Ubuntu container with: $ docker run -it ubuntu bash Share images, automate workflows, and more with a free Docker ID: https://hub.docker.com/ For more examples and ideas, visit: https://docs.docker.com/get-started/
docker info lets us inspect Docker. It will contain a lot of output but focus on the top lines which show we now have 1 container which is
stopped and 1 image.
$ docker info Client: Debug Mode: false Server: Containers: 1 Running: 0 Paused: 0 Stopped: 1 Images: 1 ...
This means Docker is successfully installed and running.
Django Hello, World
Now we will create a Django “Hello, World” project that runs locally on our computer and then move it entirely within Docker so you can see how all the pieces fit together.
The first step is to choose a location for our code. This can be anywhere on your computer, but if you are on a Mac, an easy-to-find location is the
Desktop. From the command line navigate to the
Desktop and create a
code directory for all the code examples in this book.
$ cd ~/Desktop $ mkdir code && cd code
Then create a
hello directory for this example and install Django using Pipenv which creates both a
Pipfile and a
Pipfile.lock file. Activate the virtual environment with the
$ mkdir hello && cd hello $ pipenv install django==2.2.5 $ pipenv shell (hello) $
A> If you need help installing Pipenv or Python 3 you can find more details here.
Now we can use the
startproject command to create a new Django project called
hello_project. Adding a period,
., at the end of the command is an optional step but one many Django developers do. Without the period Django adds an additional directory to the project; with the period it does not.
Finally use the
migrate command to initialize the database and start the local web server with the
(hello) $ django-admin startproject hello_project . (hello) $ python manage.py migrate (hello) $ python manage.py runserver
Assuming everything worked correctly you should now be able to navigate to see the Django Welcome page at http://127.0.0.1:8000/ in your web browser.
Now we will make a simple homepage by creating a dedicated
pages app for it. Stop the local server by typing
Control+c and then use the
startapp command appending our desired
(hello) $ python manage.py startapp pages
Django automatically installs a new
pages directory and several files for us. But even though the app has been created our
hello_project won’t recognize it until we add it to the
INSTALLED_APPS config within the
Django loads apps from top to bottom so generally speaking it’s a good practice to add new apps below built-in apps they might rely on such as
auth, and all the rest.
Note that while it is possible to simply type the name of the app,
pages, you are better off typing the full
pages.apps.PagesConfig which opens up more possibilities in configuring apps.
# hello_project/settings.py INSTALLED_APPS = [ 'django.contrib.admin', 'django.contrib.auth', 'django.contrib.contenttypes', 'django.contrib.sessions', 'django.contrib.messages', 'django.contrib.staticfiles', 'pages.apps.PagesConfig', # new ]
Now we can set the URL route for the
pages app. Since we want our message to appear on the homepage we’ll use the empty string
''. Don’t forget to add the
include import on the second line as well.
# hello_project/urls.py from django.contrib import admin from django.urls import path, include # new urlpatterns = [ path('admin/', admin.site.urls), path('', include('pages.urls')), # new ]
Rather than set up a template at this point we can just hardcode a message in our view layer at
pages/views.py which will output the string “Hello, World!”.
# pages/views.py from django.http import HttpResponse def home_page_view(request): return HttpResponse('Hello, World!')
What’s next? Our last step is to create a
urls.py file within the
pages app and link it to
home_page_view. If you are on an Mac or Linux computer the
touch command can be used from the command line to create new files. On Windows create the new file with your text editor.
(hello) $ touch pages/urls.py
Within your text editor import
path on the top line, add the
home_page_view, and then set its route to again be the empty string of
''. Note that we also provide an optional name,
home, for this route which is a best practice.
# pages/urls.py from django.urls import path from .views import home_page_view urlpatterns = [ path('', home_page_view, name='home') ]
The full flow of our Django homepage is as follows:
- when a user goes to the homepage they will first be routed to
- then routed to
- and finally directed to the
home_page_viewwhich returns the string “Hello, World!”
Our work is done for a basic homepage. Start up the local server again.
(hello) $ python manage.py runserver
If you refresh the web browser at http://127.0.0.1:8000/ it will now output our desired message.
Now it’s time to switch to Docker. Stop the local server again with
Control+c and exit our virtual environment since we no longer need it by typing
(hello) $ exit $
How do we know the virtual environment is no longer active? There will no longer be parentheses around the directory name on the command line prompt. Any normal Django commands you try to run at this point will fail. For example, try
python manage.py runserver to see what happens.
$ python manage.py runserver File "./manage.py", line 14 ) from exc ^ SyntaxError: invalid syntax
This means we’re fully out of the virtual environment and ready for Docker.
Images, Containers, and the Docker Host
A Docker image is a snapshot in time of what a project contains. It is represented by a
Dockerfile and is literally a list of instructions that must be built. A Docker container is a running instance of an image. To continue our apartment analogy from earlier, the image is the blueprint or set of plans for the apartment; the container is the actual, fully-built building.
The third core concept is the “Docker host” which is the underlying OS. It’s possible to have multiple containers running within a single Docker host. When we refer to code or processes running within Docker, that means they are running in the Docker host.
Let’s create our first
Dockerfile to see all of this theory in action.
$ touch Dockerfile
Dockerfile add the following code which we’ll walk through line-by-line below.
# Dockerfile # Pull base image FROM python:3.7 # Set environment variables ENV PYTHONDONTWRITEBYTECODE 1 ENV PYTHONUNBUFFERED 1 # Set work directory WORKDIR /code # Install dependencies COPY Pipfile Pipfile.lock /code/ RUN pip install pipenv && pipenv install --system # Copy project COPY . /code/
Dockerfiles are read from top-to-bottom when an image is created. The first instruction must be the
FROM command which lets us import a base image to use for our image, in this case Python 3.7.
Then we use the
ENV command to set two environment variables:
PYTHONUNBUFFEREDensures our console output looks familiar and is not buffered by Docker, which we don’t want
PYTHONDONTWRITEBYTECODEmeans Python wont try to write
.pycfiles which we also do not desire
Next we use
WORKDIR to set a default work directory path called
code which is where we will store our code. If we didn’t do this then each time we wanted to execute commands within our container we’d have to type in a long path. Instead Docker will just assume we mean to execute all commands from this directory.
For our dependencies we are using
Pipenv so we copy over both the
Pipfile.lock files into a
/code/ directory in Docker.
It’s worth taking a moment to explain why Pipenv creates a
Pipfile.lock, too. The concept of lock files is not unique to Python or Pipenv; in fact it is already present in package managers for most modern programming languages:
Gemfile.lock in Ruby,
composer.lock in PHP, and so on.
Pipenv was the first popular project to incorporate them into Python packaging.
The benefit of a lock file is that this leads to a deterministic build: no matter how many times you install the software packages, you’ll have the same result. Without a lock file that “locks down” the dependencies and their order, this is not necessarily the case. Which means that two team members who install the same list of software packages might have slightly different build installations.
When we’re working with Docker where there is code both locally on our computer and also within Docker, the potential for
Pipfile.lock conflicts arises when updating software packages. We’ll explore this properly in the next chapter.
Moving along we use the
RUN command to first install
Pipenv and then
pipenv install to install the software packages listed in our
Pipfile.lock, currently just Django. It’s important to add the
--system flag as well since by default Pipenv will look for a virtual environment in which to install any package, but since we’re within Docker now, technically there isn’t any virtual environment. In a way, the Docker container is our virtual environment and more. So we must use the
--system flag to ensure our packages are available throughout all of Docker for us.
As the final step we copy over the rest of our local code into the
/code/ directory within Docker. Why do we copy local code over twice, first the
Pipfile.lock and then the rest? The reason is that images are created based on instructions top-down so we want things that change often–like our local code–to be last. That way we only have to regenerate that part of the image when a change happens, not reinstall everything each time there is a change. And since the software packages contained in our
Pipfile.lock change infrequently, it makes sense to copy them over and install them earlier.
Our image instructions are now done so let’s build the image using the command
docker build . The period,
., indicates the current directory is where to execute the command. There will be a lot of output here; I’ve only included the first two lines and the last three.
$ docker build . Sending build context to Docker daemon 154.1kB Step 1/7 : FROM python:3.7 ... Step 7/7 : COPY . /code/ ---> a48b2acb1fcc Successfully built a48b2acb1fcc
Moving on we now need to create a
docker-compose.yml file to control how to run the container that will be built based upon our
$ touch docker-compose.yml
It will contain the following code.
# docker-compose.yml version: '3.7' services: web: build: . command: python /code/manage.py runserver 0.0.0.0:8000 volumes: - .:/code ports: - 8000:8000
On the top line we specify the most recent version of Docker Compose which is currently
3.7. Don’t be confused by the fact that Python is also on version
3.7 at the moment; there is no overlap between the two! It’s just a coincidence.
Then we specify which
services (or containers) we want to have running within our Docker host. It’s possible to have multiple
services running, but for now we just have one for
web. We specify how to build the container by saying, Look in the current directory
. for the
Dockerfile. Then within the container run the
command to start up the local server.
The volumes mount automatically syncs the Docker filesystem with our local computer’s filesystem. This means that we don’t have to rebuild the image each time we change a single file!
Lastly we specify the
ports to expose within Docker which will be
8000, which is the Django default.
If this is your first time using Docker, it is highly likely you are confused right now. But don’t worry. We’ll create multiple Docker images and containers over the course of this book and with practice the flow will start to make more sense. You’ll see we use very similar
docker-compose.yml files in each of our projects.
The final step is to run our Docker container using the command
docker-compose up. This command will result in another long stream of output code on the command line.
$ docker-compose up Creating network "hello_default" with the default driver Building web Step 1/7 : FROM python:3.7 ... Creating hello_web_1 ... done Attaching to hello_web_1 web_1 | Performing system checks... web_1 | web_1 | System check identified no issues (0 silenced). web_1 | September 20, 2019 - 17:21:57 web_1 | Django version 2.2.5, using settings 'hello_project.settings' web_1 | Starting development server at http://0.0.0.0:8000/ web_1 | Quit the server with CONTROL-C.
To confirm it actually worked, go back to
http://127.0.0.1:8000/ in your web browser. Refresh the page and the “Hello, World” page should still appear.
Django is now running purely within a Docker container. We are not working within a virtual environment locally. We did not execute the
runserver command. All of our code now exists and our Django server is running within a self-contained Docker container. Success!
Stop the container with
Control+c (press the “Control” and “c” button at the same time) and additionally type
docker-compose down. Docker containers take up a lot of memory so it’s a good idea to stop them in this way when you’re done using them. Containers are meant to be stateless which is why we use
volumes to copy our code over locally where it can be saved.
$ docker-compose down Removing hello_web_1 ... done Removing network hello_default
Git is the version control system of choice these days and we’ll use it in this book. First, add a new Git file with
git init, then check the
status of changes, add updates, and include a commit message.
$ git init $ git status $ git add . $ git commit -m 'ch1'
You can compare your code for this chapter with the official repository available on Github.
Docker is a self-contained environment that includes everything we need for local development: web services, databases, and more if we want. The general pattern will always be the same when using it with Django:
- create a virtual environment locally and install Django
- create a new project
- exit the virtual environment
- write a
Dockerfileand then build the initial image
- write a
docker-compose.ymlfile and run the container with
We’ll build several more Django projects with Docker so this flow makes more sense, but that’s really all there is to it. In the next chapter we’ll create a new Django project using Docker and add PostgreSQL in a separate container as our database.