Managing packages and virtual environments with Poetry¶
How to install Poetry
The recommended way to install Poetry is to use the install-poetry.py script, this will isolate
Poetry from the rest of your system. Run the following command to install Poetry.
What is Poetry?
Poetry is a tool to manage Python project dependencies and packaging. It allows you to declare the requisite libraries, which it installs and updates for you.
Setting up Poetry¶
There are two ways to set up Poetry: the first is to create a new project using Poetry, and the second is to initialise a pre-existing project.
Setting up a new project¶
Poetry is also capable of setting up a project directory for you. Here we will create a new project called poetry-demo using:
This creates a directory with the following structure.poetry-demo
├── pyproject.toml
├── README.md
├── poetry_demo
│ └── __init__.py
└── tests
└── __init__.py
Initialise existing project¶
Here we show how to initialise Poetry in a directory with an existing project. This is an interactive
process in which Poetry will prompt you to list the information required to build the pyproject.toml
file. To do this we simply run the command poetry init within the project directory.
Example of interactive process
[Bennu:~/Downloads/poetry-demo] Osiris% poetry init
This command will guide you through creating your pyproject.toml config.
Package name [poetry-demo]:
Version [0.1.0]:
Description []: This is a description of my project.
Author [User <email>, n to skip]:
License []: MIT
Compatible Python versions [^3.9]:
Would you like to define your main dependencies interactively? (yes/no) [yes]
You can specify a package in the following forms:
- A single name (requests)
- A name and a constraint (requests@^2.23.0)
- A git url (git+https://github.com/python-poetry/poetry.git)
- A git url with a revision (git+https://github.com/python-poetry/poetry.git#develop)
- A file path (../my-package/my-package.whl)
- A directory (../my-package/)
- A url (https://example.com/packages/my-package-0.1.0.tar.gz)
Search for package to add (or leave blank to continue): numpy
Found 20 packages matching numpy
Enter package # to add, or the complete package name if it is not listed:
[0] numpy
[1] numpy1
[2] mapchete-numpy
[3] hypothesis-numpy2
[4] gdal2numpy
[5] numpy_ringbuffer
[6] numpy-financial
[7] BSON-NumPy
[8] numpy2tfrecord
[9] topas2numpy
> 0
Enter the version constraint to require (or leave blank to use the latest version):
Using version ^1.22.3 for numpy
Add a package:
Would you like to define your development dependencies interactively? (yes/no) [yes] no
Generated file
[tool.poetry]
name = "poetry-demo"
version = "0.1.0"
description = "This is a description of my project."
authors = ["Tarik Zegmott <tzegmott@gmail.com>"]
license = "MIT"
[tool.poetry.dependencies]
python = "^3.9"
numpy = "^1.22.3"
[tool.poetry.dev-dependencies]
[build-system]
requires = ["poetry-core>=1.0.0"]
build-backend = "poetry.core.masonry.api"
Do you confirm generation? (yes/no) [yes]
We are going to use this method to add Poetry to your personal learning space repository. So, use
poetry init to start the process, for the dependencies add numpy.
The pyproject.toml file¶
The pyproject.toml file is the most important file to Poetry. It is this file that details the
dependencies of your project. Below is an example that was created by using the previous command.
Example pyproject.toml
Adding dependencies to a project¶
To add a new dependencies to a project, we simply add it to the pyproject.toml file under
tool.poetry.dependencies. To add numpy to our project we specify the package name and version
constraints.
Version constraint symbols
Note the caret in front of the version number, this tells poetry that when updating do not increase the left-most non-zero digit, so in the case above the versions are limited to
=1.22.3 and <2.0.0. Here is another example, ^0.2.5 is limited to >=0.2.5 and <0.3.0. The greater than and less than are also valid version constraints, but multiple requirements must be separated by a comma. For more version constraint symbols see here.
A better method of adding dependencies is to use the add command. This command automatically
resolves the version constraints of the request package and the previous packages, it also installs
the package and sub-dependencies.
Earlier when initialising Poetry, we only added numpy to the dependency list, however, plot_sin.py
also requires matplotlib. Add this to the pyproject.toml using the command above.
Developer dependencies
Poetry is also able to specify dependencies that are only required for development. These
dependencies are stated under tool.poetry.dev-dependencies within the pyproject.toml file,
these are specified in the same manner as the standard dependencies. They can be added via
the command line too by adding the -D flag to the add command.
Installing your dependencies¶
To install the dependencies defined in your project, simply run the install command.
This command will install all dependencies listed in the pyproject.toml file, include the
developer dependencies – to exclude these add the --no-dev option. If there is also a poetry.lock
file in the directory, then the versions listed within it will be the exact versions installed.
If it is not there, then the latest version allowed by the version constraints will be installed,
and a poetry.lock file will be created. The poetry.lock file should be commited to your project
repo to ensure everyone is working with the same versions of dependencies. Install the dependencies
using poetry install, then stage and commit your pyproject.toml and poetry.lock files.
Using the virtual environment¶
Poetry creates a virtual environment to ensure that the project only has access to the dependencies
listed in the pyproject.toml file. The virtual environments are created in {cache-dir}/virtualenvs,
where {cache-dir} defaults to ~/Library/Caches/pypoetry for macOS and ~/.cache/pypoetry for Unix.
There are two ways to run commands within this virtual environment. The first allows scripts or command
line tools to run within it, simply use poetry run and add either your script or a command such as
black.
The second method is to launch an interactive shell in the virtual environment using poetry shell.
Here you can launch your scripts or command line tools as normal. To exit the virtual environment, simply
type exit.
Poetry tips
- Tracing dependencies is easiest using
poetry show --tree, which displays a tree of dependencies and sub-dependencies.[Bennu:~/Downloads/poetry-demo] Osiris% poetry show --tree black 22.3.0 The uncompromising code formatter. ├── click >=8.0.0 │ └── colorama * ├── mypy-extensions >=0.4.3 ├── pathspec >=0.9.0 ├── platformdirs >=2 ├── tomli >=1.1.0 └── typing-extensions >=3.10.0.0 pendulum 1.5.1 Python datetimes made easy. ├── python-dateutil >=2.6.0.0,<3.0.0.0 │ └── six >=1.5 ├── pytzdata >=2018.3.0.0 └── tzlocal >=1.5.0.0,<2.0.0.0 └── pytz * pytest 5.4.3 pytest: simple powerful testing with Python ├── atomicwrites >=1.0 ├── attrs >=17.4.0 ├── colorama * ├── more-itertools >=4.0.0 ├── packaging * │ └── pyparsing >=2.0.2,<3.0.5 || >3.0.5 ├── pluggy >=0.12,<1.0 ├── py >=1.5.0 └── wcwidth * - Clearing the Poetry cache can be done by running
poetry cache clear --all. - Poetry is also capable of publishing projects to Pypi using
poetry publish
Next step¶
If you are happy with all that you've learnt here, then you can move on to the Docker section. Otherwise, please repeat this section before progressing.