Link Search Menu Expand Document (external link)

Simple Python packaging

Table of contents

For Python packages without binary extensions and fairly simple builds can use a modern build system instead of the classic but verbose setuptools and setup.py. The one you select doesn’t really matter that much; they all use a standard configuration language introduced in PEP 621. The PyPA’s Flit is a great option. In the future, scikit-build and meson may support this sort of configuration, enabling binary extension packages to benefit too. These PEP 621 tools currently include Hatch, PDM, Flit, Trampolim, Whey, and Setuptools. Poetry will eventually gain support in 2.0.

Binaries mostly are unsupported in existing PEP 621 tools, though better support is coming.

Classic files

These systems do not use or require setup.py, setup.cfg, or MANIFEST.in. Those are for setuptools. Unless you are using setuptools, of course, which still uses MANIFEST.in.

Selecting a backend

Backends handle metadata the same way, so the choice comes down to how you specify what files go into an SDist and extra features, like getting a version from VCS. If you don’t have an existing preference, hatchling is an excellent choice, balancing speed, configurability, and extendability.

pyproject.toml: build-system

Packages must have a pyproject.toml file that selects the backend:

[build-system]
requires = ["hatchling"]
build-backend = "hatchling.build"

pyproject.toml: project

The metadata is specified in a standards-based format:

[project]
name = "package"
description = "A great package."
readme = "README.md"
authors = [
  { name = "My Name", email = "me@email.com" },
]
maintainers = [
  { name = "Scikit-HEP", email = "scikit-hep-admins@googlegroups.com" },
]
license = { file = "LICENSE" }
requires-python = ">=3.7"

dependencies = [
  "numpy>=1.13.3",
]

classifiers = [
  "Development Status :: 4 - Beta",
  "License :: OSI Approved :: BSD License",
  "Programming Language :: Python :: 3 :: Only",
  "Programming Language :: Python :: 3.7",
  "Programming Language :: Python :: 3.8",
  "Programming Language :: Python :: 3.9",
  "Programming Language :: Python :: 3.10",
  "Topic :: Scientific/Engineering :: Physics",
]

[project.urls]
Homepage = "https://github.com/scikit-hep/package"
Documentation = "https://package.readthedocs.io/"
"Bug Tracker" = "https://github.com/scikit-hep/package/issues"
Discussions = "https://github.com/scikit-hep/package/discussions"
Changelog = "https://package.readthedocs.io/en/latest/changelog.html"

You can read more about each field, and all allowed fields, in packaging.python.org, Flit or Whey. Note that “Homepage” is special, and replaces the old url setting.

Package structure

All packages should have a src folder, with the package code residing inside it, such as src/<package>/. This may seem like extra hassle; after all, you can type “python” in the main directory and avoid installing it if you don’t have a src folder! However, this is a bad practice, and it causes several common bugs, such as running pytest and getting the local version instead of the installed version - this obviously tends to break if you build parts of the library or if you access package metadata.

This sadly is not part of the standard metadata in [project], so it depends on what backend you you use. Hatchling, Flit, PDM, and setuptools use automatic detection, while Trampolim and whey do not, requiring a tool setting.

If you don’t match your package name and import name (which you should except for very special cases), you will likely need extra configuration here.

Versioning

You can specify the version manually (as shown in the example), but the backends usually provide some automatic features to help you avoid this. Flit will pull this from a file if you ask it to. Hatchling and PDM can be instructed to look in a file or use git.

You will always need to specify that the version will be supplied dynamically with:

dynamic = ["version"]

Then you’ll configure your backend to compute the version.

Hatchling dynamic versioning

You can tell hatchling to get the version from VCS. Add hatch-vcs to your build-backend.requires, then add the following configuration:

[tool.hatch]
version.source = "vcs"
build.hooks.vcs.version-file = "src/<package>/version.py"

Or you can tell it to look for it in a file (see docs for arbitrary regex’s):

[tool.hatch]
version.path = "src/<package>/__init__.py"

(replace <package> with the package path).

Including/excluding files in the SDist

This is tool specific.

  • Hatchling info here. Hatchling uses your VCS ignore file by default, so make sure it is accurate (which is a good idea anyway).
  • Flit info here. Flit requires manual inclusion/exclusion in many cases, like using a dirty working directory.
  • PDM info here.
  • Setuptools still uses MANIFEST.in.
Warning for Flit

Flit will not use VCS (like git) to populate the SDist if you use standard tooling, even if it can do that using its own tooling. So make sure you list explicit include/exclude rules, and test the contents:

# Show SDist contents
tar -tvf dist/*.tar.gz
# Show wheel contents
unzip -l dist/*.whl

Extras

It is recommended to use extras instead of or in addition to making requirement files. These extras a) correctly interact with install requires and other built-in tools, b) are available directly when installing via PyPI, and c) are allowed in requirements.txt, install_requires, pyproject.toml, and most other places requirements are passed.

Here is an example of a simple extras:

[project.optional-dependencies]
test = [
  "pytest >=6.0",
]
mpl = [
  "matplotlib >=2.0",
]

Self dependencies can be used by using the name of the package, such as dev = ["package[test,examples]"], but this requires Pip 21.2 or newer. We recommend providing at least test, docs, and dev.