Third-Party Package Management in Python
The course is part of this learning path
This content is designed to introduce new developers to managing third-party packages using the Python package index and a tool called Pip. This course is part of a series of content designed to help you learn to program with the Python programming language.
- The Python Package Index
- The tool used to manage packages called pip
- The tool used to create isolated Python environments named venv
- Considerations to be made before installing a third-party package
This course was designed for new developers wanting to learn Python.
- That you’re familiar with the Python runtime
- That you’re familiar with the Python language syntax
- This course assumes that you’re not yet familiar with how to install and manage third-party Python software
Using packages from the Python package index allows us to gain new functionality that we didn’t need to develop. There’s a lot of value to be gained by having near-instant access to so much code. However, once we install a package from the index we’ve also introduced some potential problems.
In this lesson, we’ll consider some of the developer-facing problems introduced by using third-party packages. First thing you need to know when working with third-party packages is where they’re supposed to exist on your operating system’s file system.
The Python runtime includes a directory named site-packages that is used to store third-party code. The runtime checks this directory for Python packages and modules. Which means if we download code into this directory we can import the packages and or modules into our code using the import syntax.
The site-packages directory is going to be in a different file-system location under different circumstances. For example, it’s a different default location for Windows, Linux, macOS, etc. It can also be configured in different ways. Though these are outside the scope for this course. Just know that the exact location of the site-packages directory will be specific to your configuration.
Because site-packages directory is the default directory for third-party code, that’s where pip installs distribution packages from the index. When pip adds the contents of a package to this directory it becomes available to the Python runtime. This allows us to import and use the code from the package in our code.
Okay, so far - so good. We can use pip to install code to the site-packages directory; making it available to the Python runtime. Where are the problems I mentioned? The problems stem from the fact that there can be multiple versions of a single package. When the code inside a distribution package changes it gets a new version.
Imagine that you install the latest version of a package. You import a module and use it to perform some task. You get everything working as intended. A month goes by and there’s a new version. You install it and your code is now broken. Because the code in the package changed - and now your code needs to be updated accordingly.
This is the first problem introduced by package versions. External code changes can impact our code in significant ways. Code changes between different versions of a package might require that we change our code to match the changes in the external code. The greater the distance between versions, the more likely there will be changes needed. Because the more changes that occur the more likely that a change will impact our code.
So, that’s one problem. The second problem introduced by package versions is that we can end up with multiple Python apps that use different versions of the same package. Imagine that you have two Python applications and each requires a different version of the same package. Since third-party packages are stored in the site-packages directory we can only have one version at a time.
These two problems can introduce challenges to developers. Fortunately, there are some solutions used to manage these challenges. Let’s break down the solutions to these two problems.
To ensure that we can control which version of a package we use, pip provides mechanisms for installing specific versions as well as for determining which versions are allowed to be installed during updates. We’ll cover how to use this in another lesson. For now, just know that pip provides mechanisms for installing specific versions of a package.
So, using pip to install specific package versions solves our first problem. Because we can update to new versions and determine if everything is working correctly. If so, we keep the updated package; if not we can roll it back to the previously working version.
What about our second problem which is introduced by requiring conflicting package versions? Multiple Python applications using the same version of Python - all share the same site-packages directory. This becomes a problem when different applications require different versions of the same package. Updating to one version will break one app, and updating to the other version breaks the other.
Modern versions of Python include a module used to solve this problem. The module is called venv which is short for virtual environment. Here’s what Python’s documentation says about the venv module:
“The venv module provides support for creating lightweight “virtual environments” with their own site directories, optionally isolated from system site directories. Each virtual environment has its own Python binary (which matches the version of the binary that was used to create this environment) and can have its own independent set of installed Python packages in its site directories.”
Let’s break that down further. The venv module is used to create an isolated project. Where each project contains its own site-packages directory. That means that we can update a project’s packages independently.
We’ll cover how to use the venv module in another lesson. However, knowing that its purpose is to allow us to manage packages for each project is enough for now. Alright, let’s stop here and recontextualize this before moving forward.
Having thousands of packages available to solve a wide range of problems is amazing. It can make it much easier to solve so many challenges. However, these packages will change and evolve. And those changes have implications for the code that uses these packages.
We identified two problems stemming from the existence of multiple versions of the same package.
The first is that: code changes between different versions of a package might require that we change our code to match. Imagine a function from a third-party module being used in our code and then that function’s parameters change. Resulting in our code-breaking until we update the function calls.
Pip helps to mitigate the effects of third-party code changes by enabling us to install specific versions of a package and determine which versions are allowed to be installed during an update. The second problem stemming from the existence of multiple versions of the same package is that multiple projects might require different versions of the same package.
The venv module is used to help with this problem by creating an isolated Python environment; each virtual environment contains its own site-packages directory. Which allows us to install packages into isolated environments for each project.
We’ll cover this all more in-depth in another lesson. However, for now, this seems like a natural stopping point. Here are your key takeaways for this lesson:
- Site-packages is the name of the directory where third-party packages are installed.
- Pip helps to mitigate the effects of third-party code changes by enabling us to install specific versions of a package and determine which versions are allowed to be installed during an update.
- The venv module is used to create isolated Python environments.
Alright, that's all for this lesson. Thanks so much for watching. And I’ll see you in another lesson!
Ben Lambert is a software engineer and was previously the lead author for DevOps and Microsoft Azure training content at Cloud Academy. His courses and learning paths covered Cloud Ecosystem technologies such as DC/OS, configuration management tools, and containers. As a software engineer, Ben’s experience includes building highly available web and mobile apps. When he’s not building software, he’s hiking, camping, or creating video games.