Repositories
Repositories #
Poetry supports the use of PyPI and private repositories for discovery of packages as well as for publishing your projects.
By default, Poetry is configured to use the PyPI repository, for package installation and publishing.
So, when you add dependencies to your project, Poetry will assume they are available on PyPI.
This represents most cases and will likely be enough for most users.
Private Repository Example #
Installing from private package sources #
By default, Poetry discovers and installs packages from PyPI. But, you want to install a dependency to your project for a simple API repository? Let’s do it.
First, configure the package source as a supplemental (or explicit) package source to your project.
poetry source add --priority=supplemental foo https://pypi.example.org/simple/
Then, assuming the repository requires authentication, configure credentials for it.
poetry config http-basic.foo <username> <password>
Once this is done, you can add dependencies to your project from this source.
poetry add --source foo private-package
Publishing to a private repository #
Great, now all that is left is to publish your package. Assuming you’d want to share it privately with your team, you can configure the Upload API endpoint for your publishable repository.
poetry config repositories.foo https://pypi.example.org/legacy/
If you need to use a different credential for your package source, then it is recommended to use a different name for your publishing repository.
poetry config repositories.foo-pub https://pypi.example.org/legacy/
poetry config http-basic.foo-pub <username> <password>
Now, all the is left is to build and publish your project using the
publish
.
poetry publish --build --repository foo-pub
Package Sources #
By default, Poetry is configured to use the Python ecosystem’s canonical package index PyPI.
pypi
,
package sources are local to a project and must be configured within the project’s
pyproject.toml
file. This is not the same configuration used
when publishing a package.Project Configuration #
These package sources may be managed using the source
command for
your project.
poetry source add foo https://foo.bar/simple/
This will generate the following configuration snippet in your
pyproject.toml
file.
[[tool.poetry.source]]
name = "foo"
url = "https://foo.bar/simple/"
priority = "primary"
If priority
is undefined, the source is considered a primary source that takes precedence over PyPI, secondary, supplemental and explicit sources.
Package sources are considered in the following order:
- default source,
- primary sources,
- implicit PyPI (unless disabled by another default source or configured explicitly),
- secondary sources (DEPRECATED),
- supplemental sources.
Explicit sources are considered only for packages that explicitly indicate their source.
Within each priority class, package sources are considered in order of appearance in pyproject.toml
.
If you want to change the priority of PyPI, you can set it explicitly, e.g.
poetry source add --priority=primary PyPI
If you prefer to disable PyPI completely, you may choose to set one of your package sources to be the default or configure PyPI as explicit source.
Default Package Source #
By default, Poetry configures PyPI as the default package source for your
project. You can alter this behaviour and exclusively look up packages only from the configured
package sources by adding a single source with priority = "default"
.
poetry source add --priority=default foo https://foo.bar/simple/
In a future version of Poetry, PyPI will be disabled automatically
if there is at least one custom source configured with another priority than explicit
.
If you are using custom sources in addition to PyPI, you should configure PyPI explicitly
with a certain priority, e.g.
poetry source add --priority=primary PyPI
This way, the priority of PyPI can be set in a fine-granular way.
Secondary Package Sources (DEPRECATED) #
Deprecated in 1.5.0
If package sources are configured as secondary, all it means is that these will be given a lower priority when selecting compatible package distribution that also exists in your default and primary package sources. If the package source should instead be searched only if higher-priority repositories did not return results, please consider a supplemental source instead.
You can configure a package source as a secondary source with priority = "secondary"
in your package
source configuration.
poetry source add --priority=secondary https://foo.bar/simple/
There can be more than one secondary package source.
Supplemental Package Sources #
Introduced in 1.5.0
Package sources configured as supplemental are only searched if no other (higher-priority) source yields a compatible package distribution. This is particularly convenient if the response time of the source is high and relatively few package distributions are to be fetched from this source.
You can configure a package source as a supplemental source with priority = "supplemental"
in your package
source configuration.
poetry source add --priority=supplemental https://foo.bar/simple/
There can be more than one supplemental package source.
Explicit Package Sources #
Introduced in 1.5.0
If package sources are configured as explicit, these sources are only searched when a package configuration explicitly indicates that it should be found on this package source.
You can configure a package source as an explicit source with priority = "explicit"
in your package source configuration.
poetry source add --priority=explicit foo https://foo.bar/simple/
There can be more than one explicit package source.
Package Source Constraint #
All package sources (including secondary and possibly supplemental sources) will be searched during the package lookup process. These network requests will occur for all sources, regardless of if the package is found at one or more sources.
In order to limit the search for a specific package to a particular package repository, you can specify the source explicitly.
poetry add --source internal-pypi httpx
This results in the following configuration in pyproject.toml
:
[tool.poetry.dependencies]
...
httpx = { version = "^0.22", source = "internal-pypi" }
[[tool.poetry.source]]
name = "internal-pypi"
url = ...
priority = ...
"explicit"
.
If a repository is configured to be the source of a package, it will be the only source that is considered for that package
and the repository priority will have no effect on the resolution.Package source
keys are not inherited by their dependencies.
In particular, if package-A
is configured to be found in source = internal-pypi
,
and package-A
depends on package-B
that is also to be found on internal-pypi
,
then package-B
needs to be configured as such in pyproject.toml
.
The easiest way to achieve this is to add package-B
with a wildcard constraint:
poetry add --source internal-pypi package-B@*
This will ensure that package-B
is searched only in the internal-pypi
package source.
The version constraints on package-B
are derived from package-A
(and other client packages), as usual.
If you want to avoid additional main dependencies,
you can add package-B
to a dedicated dependency group:
poetry add --group explicit --source internal-pypi package-B@*
Supported Package Sources #
Python Package Index (PyPI) #
Poetry interacts with PyPI via its JSON API. This is used to retrieve a requested package’s versions, metadata, files, etc.
If you want to explicitly select a package from PyPI you can use the --source
option with the add
command, like shown below.
poetry add --source pypi httpx@^0.22.0
This will generate the following configuration snippet in your pyproject.toml
file.
httpx = {version = "^0.22.0", source = "pypi"}
priority = "default"
, The implicit pypi
source will
be disabled and not used for any packages.Simple API Repository #
Poetry can fetch and install package dependencies from public or private custom repositories that implement the simple repository API as described in PEP 503.
These package sources may be configured via the following command in your project.
poetry source add testpypi https://test.pypi.org/simple/
/simple/
. This is important when configuring
PEP 503 compliant package sources.In addition to PEP 503, Poetry can also handle simple API repositories that implement PEP 658 (Introduced in 1.2.0). This is helpful in reducing dependency resolution time for packages from these sources as Poetry can avoid having to download each candidate distribution, in order to determine associated metadata.
Why does Poetry insist on downloading all candidate distributions for all platforms when metadata is not available?
The need for this stems from the fact that Poetry’s lock file is platform-agnostic. This means, in order to resolve dependencies for a project, Poetry needs metadata for all platform specific distributions. And when this metadata is not readily available, downloading the distribution and inspecting it locally is the only remaining option.
Single Page Link Source #
Introduced in 1.2.0
Some projects choose to release their binary distributions via a single page link source that partially follows the structure of a package page in PEP 503.
These package sources may be configured via the following command in your project.
poetry source add jax https://storage.googleapis.com/jax-releases/jax_releases.html
Publishable Repositories #
Poetry treats repositories to which you publish packages as user specific and not project specific configuration unlike package sources. Poetry, today, only supports the Legacy Upload API when publishing your project.
These are configured using the config
command, under the
repositories
key.
poetry config repositories.testpypi https://test.pypi.org/legacy/
test.pypi.org
) as
well as the path (/legacy
) are different to it’s simple API (https://test.pypi.org/simple
).Configuring Credentials #
If you want to store your credentials for a specific repository, you can do so easily:
poetry config http-basic.foo <username> <password>
If you do not specify the password you will be prompted to write it.
To publish to PyPI, you can set your credentials for the repository named pypi
.
Note that it is recommended to use API tokens when uploading packages to PyPI. Once you have created a new token, you can tell Poetry to use it:
poetry config pypi-token.pypi <my-token>
If you still want to use your username and password, you can do so with the following
call to config
.
poetry config http-basic.pypi <username> <password>
You can also specify the username and password when using the publish
command
with the --username
and --password
options.
If a system keyring is available and supported, the password is stored to and retrieved from the keyring. In the above example, the credential will be stored using the name poetry-repository-pypi
. If access to keyring fails or is unsupported, this will fall back to writing the password to the auth.toml
file along with the username.
Keyring support is enabled using the keyring library. For more information on supported backends refer to the library documentation.
Poetry will fallback to Pip style use of keyring so that backends like Microsoft’s artifacts-keyring get a chance to retrieve valid credentials. It will need to be properly installed into Poetry’s virtualenv, preferably by installing a plugin.
If you are letting Poetry manage your virtual environments you will want a virtualenv
seeder installed in Poetry’s virtualenv that installs the desired keyring backend
during poetry install
. To again use Azure DevOps as an example: azure-devops-artifacts-helpers
provides such a seeder. This would of course best achieved by installing a Poetry plugin
if it exists for you use case instead of doing it yourself.
Alternatively, you can use environment variables to provide the credentials:
export POETRY_PYPI_TOKEN_PYPI=my-token
export POETRY_HTTP_BASIC_PYPI_USERNAME=<username>
export POETRY_HTTP_BASIC_PYPI_PASSWORD=<password>
See Using environment variables for more information on how to configure Poetry with environment variables.
If your password starts with a dash (e.g. randomly generated tokens in a CI environment), it will be parsed as a command line option instead of a password. You can prevent this by adding double dashes to prevent any following argument from being parsed as an option.
poetry config -- http-basic.pypi myUsername -myPasswordStartingWithDash
Certificates #
Custom certificate authority and mutual TLS authentication #
Poetry supports repositories that are secured by a custom certificate authority as well as those that require certificate-based client authentication. The following will configure the “foo” repository to validate the repository’s certificate using a custom certificate authority and use a client certificate (note that these config variables do not both need to be set):
poetry config certificates.foo.cert /path/to/ca.pem
poetry config certificates.foo.client-cert /path/to/client.pem
The value of certificates.<repository>.cert
can be set to false
if certificate verification is
required to be skipped. This is useful for cases where a package source with self-signed certificates
are used.
poetry config certificates.foo.cert false
Caches #
Poetry employs multiple caches for package sources in order to improve user experience and avoid duplicate network requests.
The first level cache is a Cache-Control header based cache for almost all HTTP requests.
Further, every HTTP backed package source caches metadata associated with a package once it is fetched or generated. Additionally, downloaded files (package distributions) are also cached.
Debugging Issues #
If you encounter issues with package sources, one of the simplest steps you might take to debug an issue is rerunning
your command with the --no-cache
flag.
poetry --no-cache add pycowsay
If this solves your issue, you can consider clearing your cache using the cache
command.
Alternatively, you could also consider enabling very verbose logging -vvv
along with the --no-cache
to see network
requests being made in the logs.