Premise
The goal of this snippet is to describe the creation and use of a python project structured in a way that is compatible with the src layout used by many frameworks and tools.
Project setup
There are several ways to initialize the project. The direct and manual way does not require dependencies at this stage, while relying on an external tool like PyScaffold can achieve the same result by letting the library create the basic structure.
Manual setup
Create a folder with the name of the project, for example <project>.
Inside this folder, the most important files to create are:
<project>
├── README.md # Project description
├── pyproject.toml # Project configuration
├── tox.ini # tox configuration (optional)
│
├── docs # Folder containing the documentation
│ ├── _static # Documentation static files
│ ├── conf.py # Documentation configuration
│ ├── index.rst # Documentation index
│ └── requirements.txt # Documentation dependencies
│
├── tests # Folder containing the tests
│ └── conftest.py # Test configuration
│
└── src # Folder containing the source code
└── <project>
├── __main__.py # File that will be executed when the package is called as a script
└── __init__.py # File that initializes the package
We will see later how to configure the various files.
PyScaffold setup
PyScaffold is a tool that allows you to create the basic structure of a python project. It can be installed with pip:
pip install --upgrade pyscaffold
# or with pipx
pipx install pyscaffold
# o with conda
conda install -c conda-forge pyscaffold
After installing it, simply run the putup command to create the basic project structure:
putup <project>
All the generated structure can then be modified at will. See PyScaffold for more information.
Configuration
After creating the project structure, some files need to be configured to be able to use it. Obviously, the choices to be made depend a lot on the nature and objectives of the project, so I will limit myself to describing fairly generic configurations that should be suited to most cases.
pyproject.toml
The pyproject.toml file is the main project configuration file.
Introduced with PEP518 and later extended, it is a standard component that defines dependencies, metadata, and requirements in order to build and distribute the project.
For more information, see PEP518 and PEP621, as well as the pip page.
Its structure could be as follows:
# Build system configuration
# In this case, we use setuptools and setuptools_scm
[build-system]
requires = ["setuptools>=46.1.0", "setuptools_scm[toml]>=5"]
build-backend = "setuptools.build_meta"
# Project metadata
# The obligatory fields are 'name' and 'version'
[project]
name = "<project>"
version = "0.0.1" # Hardcoded Version of the project hardcoded
# or
# dynamic = ["version"] # In this case, the section '[tool.setuptools.dynamic]' is necessary
description = "Add a short description here!"
readme = "README.md"
requires-python = ">=3.8" # Minimum python version required
license.file = "LICENSE"
authors = [{ name = "<author>", email = "<author>@mail.com" }]
maintainers = [{ name = "<maintainer>", email = "<maintainer>@mail.com" }]
keywords = ["python"] # Project keywords
classifiers = [ # Project classifiers (see https://pypi.org/classifiers/)
"Development Status :: 4 - Beta",
"Programming Language :: Python",
]
# All the dependencies of the project needed for it to run
# In this case, importlib-metadata (any version), PyYAML (versione 5.x), requests (versione < 3) e subprocess32 (only for python < 3.9)
dependencies = [
"importlib-metadata",
'PyYAML ~= 5.0',
'requests[security] < 3',
'subprocess32; python_version < "3.2"',
]
# All the optional dependencies of the project
# They can be installed with `pip install <project>[<dependency>]`
# e.g. `pip install <project>[testing]`
[project.optional-dependencies]
testing = ["pytest", "pytest-cov"]
linting = ["pylint", "black", "mypy", "isort"]
# URLs
[project.urls]
Homepage = "https://tendto.github.io/dasaturn/"
Documentation = "https://tendto.github.io/dasaturn/api/modules"
Repository = "https://github.com/TendTo/dasaturn.git"
Changelog = "https://github.com/TendTo/dasaturn/blob/main/.github/CHANGELOG.md"
# Allows the definition of scripts that can be run directly from the terminal.
# In the example, `fibonacci` is a script that runs the `run` function of the `skeleton` module of the `<project>` package
# e.g. `fibonacci 10`
[project.scripts]
fibonacci = "<project>.skeleton:run"
# Setuptool configuration (build system)
[tool.setuptools]
include-package-data = true # Include non-python files in the package
zip-safe = false # Do not compress the package using zip (deprecated)
package-dir = { "" = "src" } # Directory containing the package
packages = ["<project>"] # Package to include in the distribution
# If you want to use a dynamic version, you need to add this section
# and specify the '__version__' field in the '__init__.py' file of the package
# [tool.setuptools.dynamic]
# version = { attr = "<project>.__version__" }
# Pytest configuration (test runner)
[tool.pytest.ini_options]
minversion = "6.0" # Minimum version of pytest
addopts = [ # Additional options for pytest
"--cov=<project>", # Test coverage <project>
"--cov-report=term-missing", # Show coverage in the terminal
"--cov-report=html", # Show coverage in html format
"--cov-fail-under=80", # Fail if the coverage is less than 80%
"--verbose", # Show verbose output
]
testpaths = ["tests"] # Test paths
# Black configuration (code formatter)
[tool.black]
target-version = ['py38', 'py39', 'py310', 'py311'] # Python versions to target
line-length = 120 # Maximum line length
include_trailing_comma = false # Add a trailing comma to the last element of a list
include = '(src|tests)\/.*\.py' # Files to include
# Isort configuration (import sorter)
[tool.isort]
profile = "black" # Use the black profile to avoid conflicts between isort and black
# Pylint configuration (code linter)
[tool.pylint.MASTER]
fail-under = '10.0' # Minimum score to pass
[tool.pylint.'MESSAGES CONTROL']
disable = ["missing-module-docstring"] # Disable the warning for missing module docstrings
[tool.pylint.format]
max-line-length = 120 # Maximum line length
tox.ini (optional)
Tox is a tool that allows you to automate the build and testing of a project. Its use ensures that the project is compatible with all versions of python and all operating systems that you want to support. All commands are executed in a virtual environment, isolated from the rest of the system.
To install it, run:
pip install --upgrade tox
# or with pipx
pipx install tox
The configuration file is tox.ini and the structure could be as follows:
# Global configuration
[tox]
requires = tox>=4 # Minimum version of tox
envlist = py{38,39,310,311}-{test,lint} # Default environments to run
isolated_build = True # Isolated build
# Default configuration for all environments
[testenv]
passenv =
SETUPTOOLS_*
# Test environment that supports python 3.8, 3.9, 3.10, and 3.11
[testenv:py{,38,39,310,311}-test]
description = Invoke pytest to run automated tests
setenv =
TOXINIDIR = {toxinidir} # Directory where the tox.ini file is located
passenv = # Environment variables to pass to the test environment
HOME
SETUPTOOLS_*
extras = testing # Install optional dependencies 'testing' (see pyproject.toml)
commands = # Commands to run
pytest {posargs}
# Linting environment that supports python 3.8, 3.9, 3.10, and 3.11
[testenv:py{,38,39,310,311}-lint]
description = Perform static analysis and style checks
extras = # Install optional dependencies 'linting' and 'testing' (see pyproject.toml)
linting
testing
setenv = # Environment variables to pass to the linting environment
PATHS = {toxinidir}/src {toxinidir}/tests
commands = # Commands to run
black --check {posargs:{env:PATHS}}
pylint {posargs:{env:PATHS}}
mypy {posargs:{env:PATHS}}
isort --check-only --diff {posargs:{env:PATHS}}
# Build environment
[testenv:{build,clean}]
description =
build: Build the package in isolation according to PEP517, see https://github.com/pypa/build
clean: Remove old distribution files and temporary build artifacts (./build and ./dist)
skip_install = True # Do not install the package, just build it
changedir = {toxinidir} # Change the working directory
deps = # Install the dependencies
build: build[virtualenv]
commands =
clean: python -c 'import shutil; [shutil.rmtree(p, True) for p in ("build", "dist", "docs/_build")]'
clean: python -c 'import pathlib, shutil; [shutil.rmtree(p, True) for p in pathlib.Path("src").glob("*.egg-info")]'
build: python -m build {posargs}
# By default, both `sdist` and `wheel` are built.
# If the sdist is too large or you don't want to make it available, consider running: `tox -e build -- --wheel`
# Documentation environment
[testenv:{docs,doctests,linkcheck}]
description =
docs: Invoke sphinx-build to build the docs
doctests: Invoke sphinx-build to run doctests
linkcheck: Check for broken links in the documentation
setenv = # Environment variables to pass to the documentation environment
DOCSDIR = {toxinidir}/docs
BUILDDIR = {toxinidir}/docs/_build
docs: BUILD = html
doctests: BUILD = doctest
linkcheck: BUILD = linkcheck
deps =
-r {toxinidir}/docs/requirements.txt
commands = # Commands to run
sphinx-build --color -b {env:BUILD} -d "{env:BUILDDIR}/doctrees" "{env:DOCSDIR}" "{env:BUILDDIR}/{env:BUILD}" {posargs}
# Publish environment
[testenv:publish]
description =
Publish the package you have been developing to a package index server.
By default, it uses testpypi. If you really want to publish your package
to be publicly accessible in PyPI, use the `-- --repository pypi` option.
skip_install = True
changedir = {toxinidir}
passenv = # Environment variables to pass to the publish environment (e.g. secrets)
TWINE_USERNAME
TWINE_PASSWORD
TWINE_REPOSITORY
TWINE_REPOSITORY_URL
deps = twine # Install the dependencies
commands = # Commands to run. 'testpypi' is the default repository
python -m twine check dist/*
python -m twine upload {posargs:--repository {env:TWINE_REPOSITORY:testpypi}} dist/*
Development
The following are the most common operations to perform during the development of the project. Both the manual variant and the automated one with Tox will be presented, should you want to use it.
Note
source .venv/bin/activate (linux) or .venv\Scripts\activate (windows).
Install dependencies
# Install the project locally in editable mode.
# Any changes made to the code will be automatically applied.
pip install -e .
# Install optional dependencies 'testing' and 'linting'
pip install -e .[testing,linting]
Run tests
# Install the dependencies for the tests
pip install -e .[testing]
# Run the tests (pip)
pytest
# Run the tests (tox)
tox -e py-test
# Run the tests on all supported python versions (tox)
tox -e py{38,39,310,311}-test
Run linting
# Install the dependencies for the linting
pip install -e .[linting]
# Run the linting (pip)
black --check src tests
pylint src tests
mypy src tests
isort --check-only --diff src tests
# Run the linting (tox)
tox -e py-lint
# Run the linting on all supported python versions (tox)
tox -e py{38,39,310,311}-lint
Build
# Install the dependencies for the build
pip install build
# Run the build (pip)
python -m build
# Run the build (tox)
tox -e build
Documentation
# Install the dependencies for the documentation
pip install -r docs/requirements.txt
# Run the documentation (pip)
sphinx-build --color -b html -d "docs/_build/doctrees" "docs" "docs/_build/html"
# Run the documentation (tox)
tox -e docs
Publish the project
# Install the dependencies for the publication
pip install twine
# Run the publication (pip)
python -m twine upload dist/*
# Run the publication (tox)
tox -e publish
CI
To automate the build and test process, you can use a CI service like GitHub Actions.
To configure it, simply create .github/workflows/<ci>.yml files with a series of steps that the runner will execute.
Here are some examples that use Tox, but can be easily adapted to only use pip.
name: "Lint Python code"
on:
push:
branches:
- main
pull_request:
jobs:
lint:
runs-on: ubuntu-latest
strategy:
matrix:
os: [ubuntu-latest, macos-latest, windows-latest]
python-version: ["3.8", "3.9", "3.10", "3.11"]
steps:
- uses: actions/checkout@v4
- uses: actions/setup-python@v4
with:
python-version: ${{ matrix.python-version }}
- name: Install dependencies
run: |
python -m pip install --upgrade pip
python -m pip install tox
- name: Test with tox
run: tox -e py-lint
name: "Test Python code"
on:
push:
branches:
- main
pull_request:
jobs:
test:
runs-on: ubuntu-latest
strategy:
matrix:
os: [ubuntu-latest, macos-latest, windows-latest]
python-version: ["3.8", "3.9", "3.10", "3.11"]
steps:
- uses: actions/checkout@v4
- uses: actions/setup-python@v4
with:
python-version: ${{ matrix.python-version }}
- name: Install dependencies
run: |
python -m pip install --upgrade pip
python -m pip install tox
- name: Test with tox
run: tox -e py-test
name: "Docs test, build and deploy"
on:
push:
branches:
- main
# Sets permissions of the GITHUB_TOKEN to allow deployment to GitHub Pages
permissions:
contents: read
pages: write
id-token: write
# Allow one concurrent deployment
concurrency:
group: "pages"
cancel-in-progress: true
jobs:
test:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: actions/checkout@v4
- uses: actions/setup-python@v4
with:
python-version: "3.11"
- name: Install dependencies
run: |
python -m pip install --upgrade pip
python -m pip install tox
- name: Check all links
run: |
tox -e linkcheck
- name: Doc test
run: |
tox -e doctests
docs:
needs: test
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: ammaraskar/sphinx-action@master
with:
docs-folder: "docs/"
- name: Setup Pages
uses: actions/configure-pages@v3
- name: Upload artifact
uses: actions/upload-pages-artifact@v2
with:
path: docs/_build/html/
- name: Deploy to GitHub Pages
id: docs
uses: actions/deploy-pages@v2
name: "Build Python package"
on:
push:
branches:
- main
pull_request:
jobs:
build:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: actions/setup-python@v4
with:
python-version: "3.11"
- name: Install dependencies
run: |
python -m pip install --upgrade pip
python -m pip install tox
- name: Test with tox
run: tox -e build
name: "Publish Python package"
on:
push:
tags:
- "*"
jobs:
publish:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: actions/setup-python@v4
with:
python-version: "3.11"
- name: Install dependencies
run: |
python -m pip install --upgrade pip
python -m pip install tox
- name: Build with tox
run: tox -e build
- name: Publish to PyPI
run: tox -e publish
env:
TWINE_USERNAME: __token__
TWINE_PASSWORD: ${{ secrets.TWINE_PASSWORD }}
TWINE_REPOSITORY: testpypi
Additional content
Code of Conduct
The code of conduct is a document that describes the rules of behavior that all project participants must follow.
It can be easily inserted into the project by adding a CODE_OF_CONDUCT.md file with the content taken from the Contributor Covenant template.
Pull Request Template
The pull request template is a document that allows those who are about to open a pull request to describe in more detail the changes they have made following a predefined schema.
To make GitHub recognize it, you need to create a .github/PULL_REQUEST_TEMPLATE.md file with the content you want.
A possible template is the following:
### Prerequisites
- [ ] I have read and understood the [contributing guide](https://github.com/<TODO>/blob/main/.github/CONTRIBUTING.rst).
- [ ] The commit message follows the [conventional commits](https://www.conventionalcommits.org/en/v1.0.0/#summary) guidelines.
- [ ] Introduced tests for the changes have been added (for bug fixes / features).
- [ ] Docs have been added/updated (for bug fixes / features).
### Description
Brief description of what this PR is about.
### (If applicable) Issue closed by this PR
- [closes #issue_number]
### Does this PR introduce a breaking change?
- [ ] Yes
- [ ] No
#### (If yes) What are the changes that might break existing applications?
Description of the changes that might break existing applications.
### Python version you are using
Python version you are using (e.g. 3.8.5).
You can find it out by running `python --version` in your terminal.
### Other information
Any other information that is important to this PR, such as inspirations, further development plans, possible issues left to be solved, etc.
Miscellaneous
Some interesting files to add to the project are:
.github/CONTRIBUTING.rst: Guide for contributors.github/CHANGELOG.md: Changelog of the project.github/ISSUE_TEMPLATE.md: Template for creating issues, which can be further divided into:.github/ISSUE_TEMPLATE/bug_report.md: Template for creating bug report issues.github/ISSUE_TEMPLATE/feature_request.md: Template for creating feature request issues
.github/AUTHORS.md: Authors of the project