Package code - Comparing options
There are multiple ways to package and distribute a Python program.
Wheel (*.whl) |
PEX (and others) |
System package ( e.g. rpm, deb, brew) |
Conda |
Self contained executable (e.g. PyInstaller) |
Container image (e.g. Docker) |
|
|---|---|---|---|---|---|---|
Description |
Bundles code into single file which can be installed with pip |
Bundles code + Python dependencies into a single, executable file |
System library package which includes code and references to other system packages |
Combines Python and system packages (except OS’ standard C library) into one package system |
Self-contained executable that includes all dependencies except external C libraries |
Complete, isolated filesystem which includes everything to run the application |
Suited for |
||||||
Libraries |
yes |
no |
no |
no |
no |
no |
Development Tools |
yes |
yes |
yes |
yes |
(yes) |
no |
End-User Applications (Non-Devs) |
no |
no |
no |
no |
yes |
yes |
Includes |
||||||
Source Code |
yes |
yes |
yes |
yes |
yes |
yes |
Compiled Files |
yes |
yes |
yes |
yes |
yes |
yes |
Contained Dependencies |
||||||
Third party Python libraries |
Can be specified, but not included |
yes |
yes |
yes |
yes |
yes |
Python interpreter |
no |
no |
Can be specified, but not included |
Can be specified, but not included |
yes |
yes |
System packages (e.g. libc) |
no |
no |
Can be specified, but not included |
yes |
no |
yes |
OS (e.g. a Linux distribution) |
no |
no |
no |
no |
no |
yes |
Notes
PEX (and others)
Tools built around zipapp (part of Python standard library)
Zipapp … no C extensions supported
These tools come from big companies (Google, Facebook, Linkedin)
A *.pex file is a complete virtual environment contains all required packages
A *.pex file can be used standalone without installing it into a different environment
System Packages
RPM: Red Hat Package Manager
DEB: Debian Package Manager
System administrator approach
Considered more stable to depend on system packages than python libraries
Conda
Many PyPi packages are available as Conda package
Community project
Makers of conda distribution usually are not the developers
Containers / Docker
Suitable if application requires a very specific OS setup
Don’t use alpine OS for deploying Python apps, standard C interpreter is not well suited for Python (should always use glibc, but Alpine uses msl)
Installing binary wheels is disabled in alpine (since binary compatibility is not given); compiling source code works mostly, though
Choose Debian image to ensure that pip installs works on your container
Upgrading alpine to properly compile a library (e.g. pandas) takes a lot of tools (installation time about 15 minutes)
GOOD if - Solution |
BAD if - Downsides |
|
|---|---|---|
Wheel |
Your system libraries and Python version are standardized across applications, so you just preinstall all the necessary ones on your server farm. Or, you are packaging a simple Python application that doesn’t need much beyond what’s available on PyPI or your private PyPI-equivalent server. |
You have a heterogeneous collection of applications requiring different versions of Python. Your application requires custom C/C++/etc. libraries that can’t be installed via pip, and that you can’t assume are pre-installed. |
PEX |
- |
- |
System package |
- |
The specific packages you need are the wrong version on the particular OS version you’re using. You want to run multiple applications with multiple dependencies on the same machine or virtual machine. This is possible in theory, but can end up being difficult due to conflicts or unavailable packages. |
Conda |
Good at supporting multiple applications with different dependencies (including Python version, and C library version) on the same machine or virtual machine. |
- |
Self contained executable |
Good option if you don’t use you any special C shared libraries beyond the minimum you’d expect on any Linux distribution |
- |
Container image |
Great solution if you need to run lots of different combinations and variants, and you need some level of isolation. |
You have multiple layers of packaging: you are likely going to have to do both Docker packaging and rely on one of the previously covered mechanisms. |
Links
PEX (and others):
Conda:
Docker: