A good way to set up your Python project using Nix, Direnv, and Poetry

A problem that I, members of the Multiverse School, and other peers have encountered is how to work with a Python project without modules installed, module versions, installed Python executables, system paths, available binaries, available libraries, environmental variables, and virtual environments. Then there’s module management and system management on top of all that, so nothing above is set in stone and must be managed using something. There’s a whole lot here. There are many problems. There are many tools. Moreover, if this were easy, it’d all be figured out, and you would already be using the tool to just get it all right. Instead, here is a guide through this imperfect situation to the most perfect situation possible, in my opinion, at time of writing. It addresses starting both new projects and adapting existing projects.

This guide provides the following:

  1. Ensuring your system can use this solution.

  2. Guides through more developer and researcher oriented OS setup steps.

  3. Explains how to initialize Nix Flakes, integrate them with Direnv, and how to integrate with Poetry

What Nix, Direnv, and Poetry are

Nix is a package manager. However, how it does this is unique and is responsible for how much attention it has received, and why it is possible to use alongside other package managers. For the hyper-technical, it is idempotent, bit reproducible, and optionally isolated or replacing base system components. So this means if two people have Nix installed on their systems, they can share a configuration between the two of them and obtain identical environments. Why this is important is that in the past this has been so arduous that attempting it has not been worthwhile. Without reproducibility, different results can happen and cause extreme sinks in time resolving any issues which arise.

Direnv runs the first approved bash script in the directory hierarchy when traversing from the local directory to the root. This is deceptively useful and should be thought of like a Swiss Army Knife. This is because oftentimes when you cd between project directories, those projects have different configurations. When you cd between those project directories Direnv can just “run stuff.” So if you need to set a variable, run tests, report if code is out of date, or anything else this is how you would do that automatically. It also handles unloading some things when you leave that directory (though test that it is doing what you want here).

Poetry, in my experience, is one of the better Python virtual environment, Python module, and Python environment variable management utilities I have come across. It manages everything about a Python project from directory structure, building, Pypi integration, the works. And it seems to do so pretty well.

System Prerequisites

  • A modern Linux distribution, or MacOS

    • Using WSL 2 to run a modern Linux distribution may be sufficient.

  • Administrator access

On Linux, ensure when run with normal privileges, the commands below result in the following:

$ sysctl kernel.unprivileged_userns_clone
kernel.unprivileged_userns_clone = 1
$ sysctl kernel.apparmor_restrict_unprivileged_userns
kernel.apparmor_restrict_unprivileged_userns = 0

Additionally, the following command should be able to run with normal privileges without error:

$ unshare --user --map-current-user --net id

System Setup

If a system has not already been set up to use Nix and Direnv, this section will guide you on how to do so for Linux systems. The directions for MacOS should be relatively similar but have not been tested.

Nix Installation

In order to install Nix, use either the official installer or one developed by Determinate Systems. By reputation, the Determinate Systems installer is more well-regarded at the time of writing.

If using MacOS 15 Sequoia, please read the advisory here.

The installers are self-guided, but if you run into trouble, you may obtain help from the following official places:

Additionally, there are a number of acknowledged unofficial spaces which may be of help:

Direnv Installation

There are two different ways to install Direnv. While it is possible to do so through Nix, this guide does not recommend doing so out of separation of concerns; if one piece breaks for any reason it shouldn’t harm the rest of your system. The first is through your system’s package manager. Using your system’s package manager is preferable because it will be managed and updated with the rest of your system and not require special attention. The other is through command line installation directly from upstream using curl -sfL https://direnv.net/install.sh | bash.

Next, you must enable Direnv to work in all your CLI shells. A few popular ones are bash and zsh.

For bash, you would run echo 'eval "$(direnv hook bash)" >> ~/.bashrc. For zsh, you would run echo 'eval "$(direnv hook zsh)" >> ~/.zshrc.

For other shells or more complex configurations, please refer to the Direnv hooks reference page.

Common Project Setup

At the risk of becoming outdated, here are two key files with many subtle issues worked out for you to add to the root of your project.

Put the following in “flake.nix”.

{
  description = "Define development dependencies.";

  inputs = {
    # Which Nix upstream package branch to track
    nixpkgs.url = "nixpkgs/nixos-unstable";
  };

  # What results we're going to expose
  outputs = { self, nixpkgs }:
    let
      # What packages and system functions we'll use
      pkgs = import nixpkgs {  system = "x86_64-linux"; };
      # What version of Python we're going to explicitly track
      # NOTE: This doesn't explicitly track with what is in packages and so
      # will break at some point in the future.
      python_name = "python3.12";
      python = pkgs.python312;
    in {

      # Declare what packages we need as a record. The use as a record is
      # needed because, without it, the data contained within can't be
      # referenced in other parts of this file.
      devShells.x86_64-linux.default = pkgs.mkShell rec {
        packages = [
          pkgs.python3Full
          (pkgs.poetry.override { python3 = python; })
          pkgs.direnv
          pkgs.gcc-unwrapped
          pkgs.stdenv
          # NOTE: Put additional packages you need in this array. Packages may be found by looking them up in
          # https://search.nixos.org/packages
        ];

        # Getting the library paths needed for Python to be put into
        # LD_LIBRARY_PATH
        pythonldlibpath = "${pkgs.stdenv.cc.cc.lib}/lib:${pkgs.stdenv.cc.cc.lib.outPath}/lib:${pkgs.lib.makeLibraryPath packages}";

        # Run the following on the shell, which builds up LD_LIBRARY_PATH.
        shellHook = ''
        export LD_LIBRARY_PATH="${pythonldlibpath}"
        '';

      };
  };
}

Put the following in “.envrc”

# Modified from the Direnv wiki entry for Poetry
# It has been adjusted to check for files more thoroughly and if they are missing request corrective action.
deploy_poetry() {
  PYPROJECT_TOML="${PYPROJECT_TOML:-pyproject.toml}"
  if [[ ! -f "$PYPROJECT_TOML" ]]; then
      log_status "No pyproject.toml found. Run \`poetry init\` manually to create a \`$PYPROJECT_TOML\` first."
      return
  fi

  if [[ -d ".venv" ]]; then
      VIRTUAL_ENV="$(pwd)/.venv"
  else
      VIRTUAL_ENV=$(poetry env info --path 2>/dev/null ; true)
  fi

  if [[ -z $VIRTUAL_ENV || ! -d $VIRTUAL_ENV ]]; then
      log_status "No virtual environment exists. Executing \`poetry install\` to create one."
      poetry install
      VIRTUAL_ENV=$(poetry env info --path)
  fi

  PATH_add "$VIRTUAL_ENV/bin"
  export POETRY_ACTIVE=1  # or VENV_ACTIVE=1
  export VIRTUAL_ENV

  LOCKFILE_OK=$(poetry check --lock 2>&1 /dev/null ; true)
  if [[ -z LOCKFILE_OK ]]; then
    poetry lock --no-update
  fi

  poetry install --sync
}

# Install packages from Nix and setup some of the development environment
use flake

# Install Python specific modules modules related to Poetry
deploy_poetry

These files are non-trivial, and you are encouraged to review them afterward. Getting into their detailed mechanics right now is unnecessary and a distraction. Once they are created, you’ll need to enable Direnv to run “.envrc” by running direnv allow in the root of your project directory.

When this happens, things should just start happening. Do not be alarmed. This is intentional. What just happened is that Direnv activated the Nix Flake. The Nix Flake sets up a few environment variables and installs necessary packages. So, no matter what computer your code would need to run on, those same programs and variables would be set.

Configuring Poetry From A Blank Project

Poetry still needs to be set up. Now, you might have noticed that you never installed poetry yourself. That is because it is something that the Nix Flake just now installed for you. All you need to do now is to run poetry new, and you will be taken through a guided setup. Be sure to take a quick look through “pyproject.toml” to acquaint yourself with the file; it tracks everything Python-specific at a high level that is going on.

Note, you may need to create a blank “README.md” and set the key-value “package-mode = false” under the “[tool.poetry]” section in “pyproject.toml”.

Configuring Poetry From An Existing Project

From whatever your current tracking system is for Python modules, or lack thereof, create a “requirements.txt” file. Next, run poetry init. This is less comprehensive than poetry new and will work with existing codebases. Then, run cat requirements.txt | grep -E '^[^# ]' | cut -d= -f1 | xargs -n 1 poetry add [SO] in order to add the modules to “pyproject.toml”. Next, comb through “pyproject.toml” for the installed packages and their versions, cross-referencing your needs. This is because the versions were stripped off during the process of being added to tracking by Poetry. Once this is done, use poetry remove to remove direct tracking of packages your project does not directly and explicitly use and rely on. This helps reduce the number of needed modules and makes resolving versions easier for Poetry. Finally, rm requests.txt to make sure vestigial usages are gone.

Note, you may need to create a blank “README.md” and set the key-value “package-mode = false” under the “[tool.poetry]” section in “pyproject.toml”.

How To Manage Packages, Python Modules, And Other Details

Just because these tools combine and simplify the many aspects of project setup and system management does not mean that the work is entirely simple. In this section, I’ll highlight commands you will need to pay particular attention to and that you will need as a regular part of running your project.

Nix

  • nix flake update: Update the Nix Flake to the latest version.

  • nix flake lock: Lock the Nix Flake to the current version.

  • nix flake check: Check the Nix Flake for errors.

Poetry

  • poetry add: Add a package to the project.

  • poetry remove: Remove a package from the project.

  • poetry update: Update a package in the project.

  • poetry install --sync: Install the packages in the project to the pinned versions in “poetry.lock”.

  • poetry check: Compares “pyproject.toml” and “poetry.lock” and makes sure they are compatible.

Troubleshooting

These tools manage and hide away an enourmous amount of state. Every bit of state is something that can come back as an inexplicable surprise. Here’s a few ways to recover from things not working as you’d expect.

  • cd out of your project and back in again.

  • Check the status of “.envrc” by running direnv status.

  • Nix flakes tend to be hit or miss when they fall out of sync with tracking in git, so make sure they’re committed in git and there are no uncommitted changes in the flake.

  • Be careful when running nix flake update as it may change the versions of packages in unexpected ways.

  • Be careful when running poetry update as it may change the versions of packages in unexpected ways.

  • Be careful when running poetry install as it may change the versions of packages in unexpected ways.

  • Be careful when running poetry sync as it may change the versions of packages in unexpected ways.

  • Poetry is opinionated. If you forget to do something like create a “README.md” or remove the need of that file, it may not work as expected. This is intentional from the developers, as it enforces consistent user behavior.

  • If you are working in a directory that has a “.envrc” file, you must run direnv allow in that directory before you can use the tools it provides.

  • If these fail and you don’t know what to do, try isolating your problem to which technology of Poetry, Direnv, or Nix is the issue and getting help from the respective community or maintainers.

Citations