Using private packages with Poetry

As Python projects grow in complexity, many development teams find themselves needing to manage proprietary code across multiple projects and teams. While public package repositories like PyPI serve the open-source community well, many organizations require private package repositories for their internal libraries and tools. In this guide, we'll explore how to effectively manage private Python packages using Poetry, a modern dependency management tool that's rapidly gaining popularity in the Python ecosystem.

Why host packages privately?

Every organization's journey toward using private package repositories usually starts with a familiar pain point. Perhaps you've found yourself copying and pasting the same utility functions across different projects, or maybe your team is struggling to maintain consistent versions of internal libraries across various applications. These are telltale signs that you've outgrown simple code sharing methods and need a more robust solution.

Private package repositories solve these challenges by providing a centralized, secure location for your proprietary code. For large corporations, this might mean hosting internal machine learning models or business-critical algorithms that give them a competitive edge. Startups might use private repositories to protect their intellectual property while still providing easy installation methods for their clients. Even individual developers often find private repositories valuable for managing code across multiple side projects or consulting engagements.

Consider a financial services company developing proprietary trading algorithms. They can't release these algorithms as open-source software, but they need to share them across multiple internal applications. A private package repository allows their development team to manage these algorithms just as easily as they would manage open-source dependencies, while maintaining strict access controls.

Enter Poetry: A modern solution to package management

If you've been in the Python ecosystem for a while, you might wonder why we need another package manager when pip has served us well for years. Poetry emerged from the real-world frustrations developers faced with traditional Python package management. While pip excels at installing packages, it wasn't designed to handle the full lifecycle of package development and publishing.

Poetry approaches Python package management holistically. When you initialize a new project with Poetry, it creates a virtual environment automatically, sparing you from the ritual of manually creating and activating virtual environments. This might seem like a small convenience, but it adds up to significant time savings across a team of developers.

The real magic of Poetry becomes apparent when you're dealing with complex dependency trees. Traditional requirements.txt files don't handle dependency conflicts well, often leading to the infamous "dependency hell." Poetry, on the other hand, uses sophisticated dependency resolution algorithms to prevent conflicts before they occur. It generates a lock file that precisely specifies every package version in your environment, ensuring that all developers on your team are working with identical dependencies.

Understanding private package pepositories

Before we dive into the practical aspects of using Poetry with private packages, it's important to understand what happens behind the scenes. A private Python package repository is more than just a storage location for wheel files – it's a specialized web server that implements Python's package repository API.

When you run a command like poetry add your-private-package, Poetry needs to know where to find the package and how to authenticate with the repository. This interaction follows a standardized protocol defined in PEP 503, known as the "Simple Repository API." The repository must provide an index page listing all available packages and separate pages for each package showing its available versions.

Downloading private packages

Let's walk through the process of setting up Poetry to work with private packages. The first step is to tell Poetry about your private repository. You'll need to configure both the repository URL and your authentication credentials. Here's where Poetry's configuration system shines – it allows you to specify these settings at the user level, so you don't need to repeat them for every project.

poetry source add private-repo https://your-repository.com/simple --priority explicit

This command adds your private repository as a package source. The explicit priority is crucial here – it tells Poetry to only check this repository for packages you specifically mark as private. This prevents unnecessary authentication attempts when installing public packages.

You'll also need to set up authentication. While you can store credentials in Poetry's configuration, I recommend using environment variables, especially in CI/CD environments:

export POETRY_HTTP_BASIC_PRIVATE_REPO_USERNAME=username
export POETRY_HTTP_BASIC_PRIVATE_REPO_PASSWORD=password

Publishing your own private packages

Publishing private packages with Poetry is just as straightforward as installing them. The process begins with configuring your upload destination:

poetry config repositories.private https://your-repository.com/legacy/

The actual process of building and publishing your package is remarkably simple:

poetry build
poetry publish --repository private

Behind these simple commands, Poetry handles several complex tasks. It builds both wheel and source distributions of your package, generates the necessary metadata, and uploads everything to your private repository. It even verifies the upload was successful before completing the process.

Choosing where to host your private packages

One of the most crucial decisions when implementing private packages is choosing where to host them. The Python ecosystem offers several options, ranging from DIY solutions to enterprise-grade hosting services. Let's explore these options to help you make an informed decision for your organization.

Self-Hosted Simple Repository

For teams with minimal requirements or those just getting started with private packages, it's possible to create your own package repository that complies with PEP 503 (the Simple Repository API). At its core, this API is surprisingly straightforward – it's essentially a static website with a specific directory structure and index files.

Here's what a basic compliant repository looks like:

/simple/
    index.html
    your-package/
        index.html
        your-package-1.0.0.tar.gz
        your-package-1.0.0-py3-none-any.whl

The index.html files contain simple HTML listings of available packages and versions. While this approach is educational and might work for small teams, it comes with significant limitations. You'll need to handle security, access control, and package uploads yourself. Additionally, scaling this solution across a growing organization can become unwieldy.

Enterprise Solutions: JFrog Artifactory

For larger organizations, especially those already invested in enterprise tooling, JFrog Artifactory provides a comprehensive solution for package management. Artifactory goes beyond simple package hosting, offering features like:

• Multi-repository management (not just Python packages)
• Fine-grained access control
• Repository replication and backup
• Integration with CI/CD pipelines
• Advanced security features and vulnerability scanning

However, Artifactory's enterprise focus comes with corresponding complexity and cost. Setting up and maintaining an Artifactory installation requires dedicated DevOps resources, and the licensing costs can be significant for smaller teams or organizations.

Modern Cloud Solutions: Envelope.dev

A newer entrant to the Python package hosting space, Envelope.dev, has emerged as an attractive middle ground between DIY solutions and enterprise platforms. It's designed specifically for Python package management, offering a more streamlined and cost-effective approach.

Envelope.dev stands out for several reasons:

First, it provides unlimited package support without tiered pricing based on repository count – a refreshing departure from traditional enterprise pricing models. This makes it particularly attractive for organizations with many small packages or those looking to modularize their codebase without worrying about hosting costs.

The platform also shines in its approach to access management. Rather than forcing you into a specific authentication scheme, Envelope.dev allows extensive customization of access credentials. This flexibility means you can implement package access patterns that match your organization's security requirements and workflows.

Setting up a new private package repository on Envelope.dev is straightforward:

# Configure the repository
poetry config repositories.upload https://app.envelope.dev/legacy/

# Set your API credentials
poetry config http-basic.upload pypi your-api-key

# Upload your package
poetry publish --repository upload

Downloading packages is equally simple:

# Add the source to your Poetry configuration
poetry source add envelope https://app.envelope.dev/simple/

# Install your private package
poetry add --source envelope your-private-package

Making the Right Choice

When selecting a hosting solution for your private packages, consider these factors:

Scale of operations: A self-hosted solution might work for a small team with a few packages, but larger organizations will benefit from managed solutions like Envelope.dev or Artifactory.

Resource availability: Consider both financial and human resources. Enterprise solutions provide comprehensive features but require significant investment in both areas. Cloud-based solutions like Envelope.dev offer a middle ground with managed infrastructure and predictable pricing.

Integration requirements: If you need to integrate with existing systems or manage multiple package types beyond Python, an enterprise solution might be necessary. However, if your focus is purely on Python packages, a specialized solution could provide a better experience.

Security requirements: While all solutions can be secured appropriately, they differ in the complexity of setting up and maintaining security configurations. Managed solutions typically provide better security out of the box, with regular updates and vulnerability scanning.

Private package best practices

While the technical setup of private packages with Poetry is straightforward, there are several organizational considerations to keep in mind. Version management becomes critically important when dealing with private packages. A version bump in an internal library can potentially affect multiple projects across your organization.

Establish clear guidelines for version numbering. Semantic versioning (major.minor.patch) provides a good framework, but you need to ensure all team members understand when to increment each number. Major version bumps should be reserved for breaking changes, minor versions for new features, and patch versions for bug fixes.

Security is another crucial consideration. Private packages often contain sensitive code, so access control is paramount. Regularly rotate authentication credentials and audit access logs. Consider implementing additional security measures such as IP whitelisting for your private repository.

Documentation often becomes a challenge with private packages. Unlike public packages where documentation can live on public hosting platforms, you'll need to ensure your private package documentation is accessible to authorized users while remaining secure. Consider setting up an internal documentation site that integrates with your organization's authentication system.

Conclusion

Private Python packages, when managed properly with Poetry, provide a robust solution for code sharing within organizations. While the initial setup requires some thought and planning, the long-term benefits in terms of code organization, version management, and deployment consistency are well worth the effort.

As you implement private packages in your organization, remember that the goal is to make development more efficient and maintainable. Start small, perhaps with a single widely-used internal library, and gradually expand your private package ecosystem as you become more comfortable with the workflow.

Poetry's elegant handling of private packages is just one example of how modern Python tools are evolving to meet the needs of professional software development. By embracing these tools and practices, we can build more maintainable and secure Python applications while keeping our proprietary code properly protected.