Of all the reasons Python is a hit with developers, one of the biggest is its broad and ever-expanding selection of third-party packages. Convenient toolkits for everything from ingesting and formatting data to high-speed math and machine learning are just an import or pip install away.
But what happens when those packages don’t play nice with each other? What do you do when different Python projects need competing or incompatible versions of the same add-ons? That’s where Python virtual environments come into play.
What are Python virtual environments?
A virtual environment is a way to have multiple, parallel instances of the Python interpreter, each with different sets of packages and different configurations. Each virtual environment contains a discrete copy of the Python interpreter, including copies of its support utilities.
The packages installed in each virtual environment are seen only in that virtual environment and no other. Even large, complex packages with platform-dependent binaries can be corralled off from each other in virtual environments.
Why use Python virtual environments?
There are a few common use cases for a virtual environment:
- You’re developing multiple projects that depend on different versions of the same packages, or you have a project that must be isolated from certain packages because of a namespace collision. This is the most standard use case.
- You’re working in a Python environment where you can’t modify the site-packages directory. This may be because you’re working in a highly controlled environment, such as managed hosting, or on a server where the choice of interpreter (or packages used in it) can’t be changed because of production requirements.
- You want to experiment with a specific combination of packages under highly controlled circumstances, for instance to test cross-compatibility or backward compatibility.
- You want to run a “baseline” version of the Python interpreter on a system with no third-party packages, and only install third-party packages for each individual project as needed.
Nothing says you can’t simply unpack a Python library into a subfolder of a project and use it that way. Likewise, you could download a standalone copy of the Python interpreter, unpack it into a folder, and use it to run scripts and packages devoted to it.
But managing such cobbled-together projects soon becomes difficult. It only seems easier to do that at first. Working with packages that have binary components, or that rely on elaborate third-party dependencies, can be a nightmare. Worse, reproducing such a setup on someone else’s machine, or on a new machine you manage, is tricky.
The best long-term solution is to use Python’s native mechanisms for creating, reproducing, and working with virtual environments.
How to use virtual environments in Python 3
Python has native tooling for virtual environments that makes the whole process quite simple. This wasn’t always the case, but now all supported versions of Python use the native virtual environment tool, venv.
Create the Python virtual environment
To create a virtual environment in a given directory, type:
python -m venv /path/to/directory
Note that you should use python3 instead of python if your system has both Python 2 and Python 3 present. On Windows, you can use py instead of python to reliably access an installed Python version. (See this article for more about using the py launcher in Windows.)
The whole process of setting up the virtual environment may take a minute or two. When it’s finished, you should have a directory with a few subdirectories in it. The most important subdirectory is bin on Unix or Scripts on Windows, which is where you’ll find the copy of the Python interpreter for the virtual environment along with its utilities.
Note that because each virtual environment contains its own copy of the Python interpreter, it can be fairly large. A Python 3.11 virtual environment will consume anywhere from 12 MB to 25 MB of disk space, depending on the operating system.
Activate the Python virtual environment
Before you can use this virtual environment, you need to explicitly activate it. Activation makes the virtual environment the default Python interpreter for the duration of a shell session.
You’ll need to use different syntax for activating the virtual environment depending on which operating system and command shell you’re using.
- On Unix or MacOS, using the bash shell:
source /path/to/venv/bin/activate - On Unix or MacOS, using the csh shell:
source /path/to/venv/bin/activate.csh - On Unix or MacOS, using the fish shell:
source /path/to/venv/bin/activate.fish - On Windows using the Command Prompt:
path\to\venv\Scripts\activate.bat - On Windows using PowerShell:
path\to\venv\Scripts\Activate.ps1
Note that the activated environment only works for the context it was activated in. For instance, if you launch two instances of PowerShell, A and B, and you only activate the virtual environment in instance A, that environment will only apply to A. It wouldn’t apply anywhere else.
Many Python IDEs will automatically detect and activate a virtual environment if one is found in the current project directory. Visual Studio Code, for instance, can do this when the Python extension is enabled. Opening a terminal inside Visual Studio Code will automatically activate the selected virtual environment. PyCharm automatically creates a virtual environment for each new project.
Configure and use the Python virtual environment
Once you’ve activated the new virtual environment, you can use the pip package manager to add and change packages for it. You’ll find pip in the Scripts subdirectory of the virtual environment on Windows, and in the bin subdirectory on Unix OSes.
If you’re already familiar with the way pip works, you’re set. It should be just the same in a virtual environment. Just make sure you’re using the instance of pip that manages packages for the virtual environment in the context where it was activated—e.g., the bash session or Windows CLI/PowerShell session. If you want to verify that you’re using the right pip and the right virtual environment, type pip -V and check that the path it displays points to a subdirectory of your virtual environment.
Note that when you want to upgrade pip in a virtual environment, it’s best to use the command python -m pip install -U pip. This ensures the upgrade process is run in such a way that Python doesn’t lock crucial files. The command pip install -U pip may not be able to complete the upgrade properly.
To use the virtual environment you created to run Python scripts, simply invoke Python from the command line in the context where you activated it. For instance, to run a script, just run python myscript.py.
With PyCharm, you can use the IDE’s own package management interface to manage the packages installed in your project.
Managing packages in Python virtual environments
When you create a new virtual environment, the pip and setuptools packages will be installed, but that’s all. You’ll need to install any other packages you want to use in the environment. For projects with complex requirements, you should keep in the root of the project a requirements.txt file that lists the requirements for the project. This way, if you need to recreate the virtual environment, you can reinstall all of the needed packages with the command pip install -r requirements.txt.
More recently, a new project metadata format has emerged for Python projects, called pyproject.toml. A pyproject.toml file contains the package requirements of the project, but also a great deal of other information about it. To install those requirements, you’d run pip install . in the same directory as the pyproject.toml file.
Note that the copies of pip and setuptools that live in a virtual environment are local to that virtual environment. Each virtual environment has its own copies, which will need to be updated and maintained independently. This is why you may get warnings about pip being out of date in some virtual environments but not others; pip has to be updated in each virtual environment separately.
Deactivating the Python virtual environment
When you’re done using the virtual environment, you can just terminate the session where you were using it. If you want to continue to work in the same session but with the default Python interpreter instead, type deactivate at the prompt. Windows users on the Command Prompt need to run deactivate.bat from the Scripts subdirectory, but Unix users and Windows users running PowerShell can simply type deactivate in any directory.
Removing the Python virtual environment
Virtual environments are self-contained. When you no longer need the virtual environment, you can just delete its directory. Just make sure you first close any running copies of Python that use the virtual environment.
Relocating the Python virtual environment
It’s tempting to assume a virtual environment can be copied and moved around along with its project. Don’t do this. Virtual environments are tied to the location of the Python installation on the system where they’re created. If you want to move the project to another system, leave out the venv directory, and recreate the venv on the target machine. Do copy and move the requirements.txt or pyproject.toml file with the project, because those files are needed to recreate the venv on the other system.
How to use virtual environments in Python 2
With Python 2, virtual environments aren’t a native feature of the language. Instead, you need to install third-party libraries to create and manage virtual environments.
The most popular and widely used of these projects is virtualenv, which handles creating the directory structure and copying the needed files into a virtual environment. To install virtualenv, just use pip install virtualenv. To create a virtual environment directory with it, type virtualenv /path/to/directory. Activating and deactivating the virtual environment works the same way as it does for virtual environments in Python 3 (see above).
Note that Python 2 should not be used for any new development. Virtual environments in Python 2, like Python 2 itself, should be used only for the maintenance of legacy projects that should eventually be migrated to Python 3.
Using Python virtual environments with Jupyter notebooks
If you’re using Jupyter notebooks (aka IPython notebooks), and you already have Jupyter installed systemwide, create your virtual environment and activate it. Then, from your virtual environment directory, run pip install ipykernel to add the needed components for IPython. Finally, run ipython kernel install —user —name=<project_name>, where project_name is a name you want to associate with that particular project. From there you should be able to launch Jupyter and switch to the IPython kernel you installed inside the virtual environment.
Upgrading Python virtual environments
When you upgrade a Python runtime on your system, virtual environments that use that version of Python aren’t automatically upgraded. That’s your responsibility. And that’s by design, because unwitting upgrades to Python versions can break their attendant packages.
If you’ve upgraded an existing Python interpreter with a minor point upgrade—e.g., from Python 3.11.1 to Python 3.11.3—you can upgrade any corresponding virtual environments easily enough. From a command prompt in the project directory, enter:
python -m venv /path/to/venv --upgrade
Don’t activate the virtual environment beforehand, or the upgrade may not work.
Alternatively, you could elect to remove the venv completely and recreate it using your requirements.txt or pyproject.toml file.
If you’ve installed a major new version of Python—e.g., you already have Python 3.10 and you now install Python 3.11 alongside it—you’ll need to create a new virtual environment that specifically uses the new major point version. Do not attempt to upgrade an existing virtual environment to a higher major point version of Python.
Copyright © 2023 IDG Communications, Inc.
