Package code - Comparing options

There are multiple ways to package and distribute a Python program.

Comparison chart (–> increase in content from left to right)

Wheel (*.whl)

PEX (and others)

System package ( e.g. rpm, deb, brew)

Conda

Self contained executable (e.g. PyInstaller)

Container image (e.g. Docker)

Description

Bundles code into single file which can be installed with pip

Bundles code + Python dependencies into a single, executable file

System library package which includes code and references to other system packages

Combines Python and system packages (except OS’ standard C library) into one package system

Self-contained executable that includes all dependencies except external C libraries

Complete, isolated filesystem which includes everything to run the application

Suited for

Libraries

yes

no

no

no

no

no

Development Tools

yes

yes

yes

yes

(yes)

no

End-User Applications (Non-Devs)

no

no

no

no

yes

yes

Includes

Source Code

yes

yes

yes

yes

yes

yes

Compiled Files

yes

yes

yes

yes

yes

yes

Contained Dependencies

Third party Python libraries

Can be specified, but not included

yes

yes

yes

yes

yes

Python interpreter

no

no

Can be specified, but not included

Can be specified, but not included

yes

yes

System packages (e.g. libc)

no

no

Can be specified, but not included

yes

no

yes

OS (e.g. a Linux distribution)

no

no

no

no

no

yes

Notes

PEX (and others)

  • Tools built around zipapp (part of Python standard library)

  • Zipapp … no C extensions supported

  • These tools come from big companies (Google, Facebook, Linkedin)

  • A *.pex file is a complete virtual environment contains all required packages

  • A *.pex file can be used standalone without installing it into a different environment

System Packages

  • RPM: Red Hat Package Manager

  • DEB: Debian Package Manager

  • System administrator approach

  • Considered more stable to depend on system packages than python libraries

Conda

  • Many PyPi packages are available as Conda package

  • Community project

  • Makers of conda distribution usually are not the developers

Containers / Docker

  • Suitable if application requires a very specific OS setup

  • Don’t use alpine OS for deploying Python apps, standard C interpreter is not well suited for Python (should always use glibc, but Alpine uses msl)

  • Installing binary wheels is disabled in alpine (since binary compatibility is not given); compiling source code works mostly, though

  • Choose Debian image to ensure that pip installs works on your container

  • Upgrading alpine to properly compile a library (e.g. pandas) takes a lot of tools (installation time about 15 minutes)

General recommendations

GOOD if - Solution

BAD if - Downsides

Wheel

Your system libraries and Python version are standardized across applications, so you just preinstall all the necessary ones on your server farm. Or, you are packaging a simple Python application that doesn’t need much beyond what’s available on PyPI or your private PyPI-equivalent server.

You have a heterogeneous collection of applications requiring different versions of Python. Your application requires custom C/C++/etc. libraries that can’t be installed via pip, and that you can’t assume are pre-installed.

PEX

-

-

System package

-

The specific packages you need are the wrong version on the particular OS version you’re using. You want to run multiple applications with multiple dependencies on the same machine or virtual machine. This is possible in theory, but can end up being difficult due to conflicts or unavailable packages.

Conda

Good at supporting multiple applications with different dependencies (including Python version, and C library version) on the same machine or virtual machine.

-

Self contained executable

Good option if you don’t use you any special C shared libraries beyond the minimum you’d expect on any Linux distribution

-

Container image

Great solution if you need to run lots of different combinations and variants, and you need some level of isolation.

You have multiple layers of packaging: you are likely going to have to do both Docker packaging and rely on one of the previously covered mechanisms.