{
"$type": "site.standard.document",
"canonicalUrl": "https://rednafi.com/python/dependency-management-redux/",
"description": "Modern Python dependency management using pip-tools, hatch, and PEP-621 for web apps and libraries with reproducible builds.",
"path": "/python/dependency-management-redux/",
"publishedAt": "2023-06-27T00:00:00.000Z",
"site": "at://did:plc:fgtm2c26vfcj74rfmeggbyqj/site.standard.publication/3mnl6f7ob462z",
"tags": [
"Python",
"DevOps"
],
"textContent": "One major drawback of Python's huge ecosystem is the significant variances in workflows\namong people trying to accomplish different things. This holds true for dependency\nmanagement as well. Depending on what you're doing with Python - whether it's building\nreusable libraries, writing web apps, or diving into data science and machine learning -\nyour workflow can look completely different from someone else's. That being said, my usual\napproach to any development process is to pick a method and give it a shot to see if it\nworks for my specific needs. Once a process works, I usually automate it and rarely revisit\nit unless something breaks.\n\nAlso, I actively try to abstain from picking up tools that haven't stood the test of time.\nIf the workflow laid out here doesn't work for you and something else does, that's\nfantastic! I just wanted to document a more modern approach to the dependency management\nworkflow that has reliably worked for me over the years. Plus, I don't want to be the person\nwho still uses [distutils] in their package management workflow and gets reprehended by\npip for doing so.\n\nDefining the scope\n\nSince the dependency management story in Python is a huge mess for whatever reason, to avoid\ngetting yelled at by the most diligent gatekeepers of the internet, I'd like to clarify the\nscope of this piece. I mainly write web applications in Python and dabble in data science\nand machine learning every now and then. So yeah, I'm well aware of how great [conda] is\nwhen you need to deal with libraries with C dependencies. However, that's not typically my\nday-to-day focus. Here, I'll primarily delve into how I manage dependencies when developing\nlarge-scale web apps and reusable libraries.\n\nIn applications, I manage my dependencies with [pip] and [pip-tools], and for libraries, my\npreferred build backend is [hatch]. [PEP-621] attempts to standardize the process of storing\nproject metadata in a pyproject.toml file, and I absolutely love the fact that now, I'll\nmostly be able to define all my configurations and dependencies in a single file. This made\nme want to rethink how I wanted to manage the dependencies without sailing against the\ncurrent recommended standard while also not getting swallowed into the vortex of conflicting\nopinions in this space.\n\nIn applications\n\nWhether I'm working on a large Django monolith or exposing a microservice via FastAPI or\nFlask, while packaging an application, I want to be able to:\n\n- Store all project metadata, linter configs, and top-level dependencies in a\n pyproject.toml file following the [PEP-621] conventions.\n- Separate the top-level application and development dependencies.\n- Generate requirements.txt and requirements-dev.txt files from the requirements\n specified in the TOML file, where the top-level and their transient dependencies will be\n pinned to specific versions.\n- Use vanilla pip to build the application hermetically from the locked dependencies\n specified in the requirements.txt files.\n\nThe goal is to simply be able to run the following command to install all the pinned\ndependencies in a reproducible manner:\n\npip-tools allows me to do exactly that. Suppose, you have an app where you're defining the\ntop-level dependencies in a canonical pyproject.toml file like this:\n\nHere, following PEP-621 conventions, we've specified the app and dev dependencies in the\nproject.dependencies and project.optional-dependencies.dev sections respectively. Now in\na virtual environment, install pip-tools and run the following commands:\n\nRunning the commands will create two lock files requirements.txt and\nrequirements-dev.txt where all the pinned top-level and transient dependencies will be\nlisted out. The contents of the requirements.txt file looks like this (truncated):\n\nSimilarly, the content of requirements-dev.txt file goes as follows (truncated):\n\nOnce the lock files are generated, you're free to build the application in however way you\nsee fit and the build process doesn't even need to be aware of the existence of pip-tools.\nIn the simplest case, you can just run pip install to build the application. Check out\nthis [fastapi-nano example] that uses the workflow explained in this section.\n\nIn libraries\n\nWhile packaging libraries, I pretty much want the same things mentioned in the application\nsection. However, the story of dependency management in reusable libraries is a bit more\nhairy. Currently, there's no standard around a lock file and I'm not aware of a way to build\nartifacts from a plain requirements.txt file. For this purpose, my preferred build backend\nis [hatch]. Mostly because it follows the latest standards formalized by the associated\nPEPs. From the FAQ section of the hatch docs:\n\n> Q: What is the risk of lock-in?\n>\n> A: Not much! Other than the plugin system, everything uses Python's established standards\n> by default. Project metadata is based entirely on [PEP-621]/[PEP-631], the build system is\n> compatible with [PEP-517]/[PEP-660], versioning uses the scheme specified by [PEP-440],\n> dependencies are defined with [PEP-508] strings, and environments use virtualenv.\n\nHowever, it doesn't support lock files yet:\n\n> The only caveat is that currently there is no support for re-creating an environment given\n> a set of dependencies in a reproducible manner. Although a standard lock file format may\n> be far off since [PEP-665] was rejected, resolving capabilities are coming to pip. When\n> that is stabilized, Hatch will add locking functionality and dedicated documentation for\n> managing applications.\n\nIn my experience, I haven't faced many issues regarding the lack of support for lock files\nwhile building reusable libraries. Your mileage may vary.\n\nNow let's say we're trying to package up a CLI that has the following source structure:\n\nThe content of cli.py looks like this:\n\nThe corresponding pyproject.toml file looks as follows:\n\nNow install hatch in your virtualenv and run the following command to create the build\nartifacts:\n\nThis will create the build artifacts in the src directory:\n\nYou can now install the local wheel file to test the build:\n\nOnce you've installed the CLI locally, you can test it by running foo-cli from your\nconsole:\n\nThis returns:\n\nYou can also build and install the CLI with:\n\nHatch also provides a hatch publish command to upload the package to PyPI. For a complete\nreference, check out how I shipped [rubric] following this workflow.\n\nFurther reading\n\n- [Using pyproject.toml in your Django project - Peter Baumgartner]\n- [TIL: pip-tools Supports pyproject.toml - Hynek Schlawack]\n\n\n\n\n[distutils]:\n https://docs.python.org/3.11/distutils/\n\n[conda]:\n https://docs.conda.io/en/latest/\n\n[pip]:\n https://pip.pypa.io/en/stable/\n\n[pip-tools]:\n https://pip-tools.readthedocs.io/en/latest/\n\n[hatch]:\n https://hatch.pypa.io/latest/\n\n[pep-621]:\n https://peps.python.org/pep-0621/\n\n[fastapi-nano example]:\n https://github.com/rednafi/fastapi-nano/blob/master/pyproject.toml\n\n[pep-631]:\n https://peps.python.org/pep-0631/\n\n[pep-517]:\n https://peps.python.org/pep-0517/\n\n[pep-660]:\n https://peps.python.org/pep-0660/\n\n[pep-440]:\n https://peps.python.org/pep-0440/\n\n[pep-508]:\n https://peps.python.org/pep-0508/\n\n[pep-665]:\n https://peps.python.org/pep-0665/\n\n[rubric]:\n https://github.com/rednafi/rubric/blob/main/pyproject.toml\n\n[using pyproject.toml in your django project - peter baumgartner]:\n https://lincolnloop.com/insights/using-pyprojecttoml-in-your-django-project/\n\n[til: pip-tools supports pyproject.toml - hynek schlawack]:\n https://hynek.me/til/pip-tools-and-pyproject-toml/",
"title": "Python dependency management redux"
}