Link Search Menu Expand Document

Simple Python packaging

Table of contents

For Python packages without binary extensions and fairly simple builds can use a modern build system instead of the classic but verbose setuptools and setup.py. The one you select doesn’t really matter that much; they all use a standard configuration language specified in PEP 621. The PyPA’s Flit is a great option. In the future, scikit-build and meson may support this sort of configuration, enabling binary extension packages to benefit too. These PEP 621 tools currently include Hatch, PDM, Flit, Trampolim, Whey, and Setuptools. Poetry will eventually gain support in 2.0.

Binaries mostly are unsupported in existing PEP 621 tools, though better support is coming.

Classic files

These systems do not use or require setup.py, setup.cfg, or MANIFEST.in. Those are for setuptools.

SDist compatibility

Some systems produce a compatibility setup.py to allow use in very old environments, but there is not much need for this - 99% of users of pure Python packages will simply get the wheel - you have to ask for the SDist, and if you do, you are probably a package manager and know what you are doing. And if you are, you need to support modern style builds now; the PyPA packages are starting to drop setup.py’s.

Selecting a backend

Backends handle metadata the same way, so the choice comes down to how you specify what files go into an SDist and extra features, like getting a version from VCS.

pyproject.toml: build-system

Packages must have a pyproject.toml file that selects the backend:

[build-system]
requires = ["hatchling>=0.7"]
build-backend = "hatchling.build"

pyproject.toml: project

The metadata is specified in a standards-based (PEP 621) format:

[project]
name = "package"
description = "A great package."
readme = "README.md"
authors = [
  { name = "My Name", email = "me@email.com" },
]
maintainers = [
  { name = "Scikit-HEP", email = "scikit-hep-admins@googlegroups.com" },
]
license = { file = "LICENSE" }
requires-python = ">=3.7"

dependencies = [
  "numpy>=1.13.3",
]

classifiers = [
  "Development Status :: 4 - Beta",
  "License :: OSI Approved :: BSD License",
  "Programming Language :: Python :: 3 :: Only",
  "Programming Language :: Python :: 3.7",
  "Programming Language :: Python :: 3.8",
  "Programming Language :: Python :: 3.9",
  "Programming Language :: Python :: 3.10",
  "Topic :: Scientific/Engineering :: Physics",
]

[project.urls]
Homepage = "https://github.com/scikit-hep/package"
Documentation = "https://package.readthedocs.io/"
"Bug Tracker" = "https://github.com/scikit-hep/package/issues"
Discussions = "https://github.com/scikit-hep/package/discussions"
Changelog = "https://package.readthedocs.io/en/latest/changelog.html"

You can read more about each field, and all allowed fields, in PEP 621, Flit or Whey. Note that “Homepage” is special, and replaces the old url setting.

Package structure

All packages should have a src folder, with the package code residing inside it, such as src/<package>/. This may seem like extra hassle; after all, you can type “python” in the main directory and avoid installing it if you don’t have a src folder! However, this is a bad practice, and it causes several common bugs, such as running pytest and getting the local version instead of the installed version - this obviously tends to break if you build parts of the library or if you access package metadata.

This sadly is not part of the standard metadata in [project], so it depends on what backend you you use. Flit, Hatch, PDM, and (experimental) setuptools use automatic detection, while Trampolim and whey do not, requiring a tool setting.

Versioning

You can specify the version manually (as shown in the example), but the backends usually provide some automatic features to help you avoid this. Flit will pull this from a file if you ask it to. PDM can be instructed to look in a file or use git. Trampolim can be instructed to use Git.

Including/excluding files in the SDist

This is tool specific. Hatchling info here. Flit info here. PDM info here. Whey info here.

Warning for Flit

Flit will not use VCS (like git) to populate the SDist if you use standard tooling, even if it can do that using its own tooling. So make sure you list explicit include/exclude rules, and test the contents:

# Show SDist contents
tar -tvf dist/*.tar.gz
# Show wheel contents
unzip -l dist/*.whl

Extras

It is recommended to use extras instead of or in addition to making requirement files. These extras a) correctly interact with install requires and other built-in tools, b) are available directly when installing via PyPI, and c) are allowed in requirements.txt, install_requires, pyproject.toml, and most other places requirements are passed.

Here is an example of a simple extras:

[project.optional-dependencies]
test = [
  "pytest >=6.0",
]
mpl = [
  "matplotlib >=2.0",
]

Self dependencies can be used by using the name of the package, such as dev = ["package[test,examples]"], but this requires Pip 21.2 or newer. We recommend providing at least test, docs, and dev.