Using Python and Docker to create a novel and scalable development environment solution with Jordan Singleton

Local development environments are essential tools which help us run code on our own machines. Good development environments help us efficiently manage our workflow as we make code changes, and they can be adapted to fit a changing stack and team. In this talk, we’ll look at shortcomings of different development environment solutions, and how we’ve attempted to solve them with Python and Docker.

The simplest solution is to just run your app on the command line - for example with python my_app.py. This is easy and works well if your setup is simple. But as you scale your stack, adding apps and developers, this approach becomes problematic. When you have more than one app to run, you can run into dependency conflicts between apps. When you add more developers, it becomes a challenge to reproduce your system configuration so that each app runs smoothly on each developer’s machine. Virtualenvs help these issues by providing a measure of isolation and reproducibility. However they only partly solve the problem; environment variables, system libraries, and other system configuration all hamper reproducibility and isolation.

A common approach to these problems is to run code in a virtual machine. This does increase the portability of an environment, but still comes with some issues. Changing dependencies and configuration over time can prompt changes in the VM system state, which can create the same reproducibility issues as before. Configuration management software can help, but comes with its own set of issues. Additionally, virtual machines come with performance penalties; shared folders have poor performance, and the VM will consume tons of memory if you have a large stack. Finally, apps running on the same VM can still suffer from lack of isolation. Conflicting system dependencies, apps that want to listen on the same port, and other issues will require workarounds.

Docker containers provide an opportunity to improve on these shortcomings. Containers are by nature isolated and portable, and they come without the performance overhead that’s associated with virtual machines. However there are still problems to solve before containers can be managed easily in a development environment workflow. How are the containers launched in a reproducible way? How are containers connected so that they can communicate as needed? What happens when we need to make changes to the stack being run? We’ve built a Python app which manages docker containers and abstracts these processes. Dusty (dusty.gc.com) knows how to run an arbitrarily complex stack based on a set of defined specs, and provides other workflow management features. We think that this approach is more portable, more scalable, and easier to use than other development environment solutions capable of running complex stacks.