Nix Runtime

Goal

The Nix setup in this repo is designed to provide a stable Linux development runtime with a narrow host boundary:

  • Nix owns Python, compiler toolchains, FFmpeg, OpenGL/Vulkan userspace, CUDA userspace, and project Python packages.

  • The host only provides the NVIDIA driver boundary:

    • libcuda.so.1

    • NVIDIA GL/EGL/Vulkan driver libraries

    • Vulkan ICD JSON

    • EGL vendor JSON

This is the closest practical approximation to “one reproducible dev shell across Linux hosts with NVIDIA drivers installed”.

The intended consumption model is library-first:

  • prefer import simple from inside the dev shell

  • treat uv run eval, datagen, and similar CLI entry points as convenience wrappers around the Python API

  • if another project integrates SIMPLE, it should usually depend on the package/runtime, not shell out to the CLI

Why The Setup Exists

The environment is more complex than a pure CPU Python project because this repo mixes:

  1. Python packaging managed by uv

  2. Native extension builds for vendored packages

  3. CUDA userspace managed by Nix

  4. NVIDIA driver userspace managed by the host

  5. Isaac Sim / CuRobo GPU-specific post-install work

The complexity does not come from Nix alone. It comes from the boundary between:

  • reproducible userspace we can manage in Nix

  • GPU driver components that remain host-specific

Supported Host Model

The intended host baseline is:

  • Linux

  • NVIDIA drivers already installed

  • Ubuntu-style driver layout is explicitly supported

  • NixOS-style /run/opengl-driver/... layout is also supported

  • WSL driver layout is partially supported through /usr/lib/wsl/lib

  • at least ~80 GiB free on /nix is strongly recommended for a first CUDA-heavy bootstrap

  • at least ~20 GiB free on the workspace filesystem is recommended for caches, extracted wheels, and virtualenvs

Current host lookup paths:

  • CUDA:

    • /run/opengl-driver/lib/libcuda.so.1

    • /usr/lib/x86_64-linux-gnu/libcuda.so.1

    • /usr/lib/x86_64-linux-gnu/nvidia/current/libcuda.so.1

    • /usr/lib/x86_64-linux-gnu/nvidia/libcuda.so.1

    • /usr/lib/wsl/lib/libcuda.so.1

  • Vulkan ICD:

    • /run/opengl-driver/share/vulkan/icd.d/nvidia_icd.x86_64.json

    • /usr/share/vulkan/icd.d/nvidia_icd.json

    • /usr/share/vulkan/icd.d/nvidia_icd.x86_64.json

    • /etc/vulkan/icd.d/nvidia_icd.json

  • EGL vendor:

    • /run/opengl-driver/share/glvnd/egl_vendor.d/10_nvidia.json

    • /usr/share/glvnd/egl_vendor.d/10_nvidia.json

    • /usr/share/glvnd/egl_vendor.d/50_nvidia.json

If an Ubuntu host keeps NVIDIA files elsewhere, update scripts/nix/common.sh.

Runtime Layout

1. Base Nix runtime

nix/runtime-base.nix

Owns:

  • Python runtime

  • wrapped compiler toolchain

  • libc handling via the wrapped compiler toolchain

  • CUDA toolkit userspace

  • FFmpeg and graphics userspace libs

Important rule:

  • do not add glibc.dev as a regular shell package to “help” C/C++ builds

  • the wrapped compiler already knows how to find libc headers in the right order

  • adding glibc.dev directly can break libstdc++’s #include_next chain for stdlib.h

2. Host NVIDIA bridge

nix/runtime-host-gpu.nix

Owns:

  • staging host NVIDIA driver libraries into .runtime-state/host-libcuda

  • exposing Vulkan/EGL host metadata into the shell

3. Shared runtime policy

scripts/nix/common.sh

Owns:

  • sourcing ~/.env when present

  • sanitizing Python-related env vars

  • defining allowed runtime library paths

  • validating the Nix/host-driver boundary

  • checking vendored dependency trees

  • detecting TORCH_CUDA_ARCH_LIST

4. Shell orchestration

scripts/nix/shell-init.sh

Owns:

  • per-shell writable cache/config dirs

  • project virtualenv path policy

  • shell-time assertions

  • optional auto-bootstrap

5. Bootstrap phases

This split is intentional:

  • Python/bootstrap should remain readable and mostly host-independent.

  • GPU/bootstrap is where CuRobo and GPU-specific work belongs.

Runtime Invariants

The shell/bootstrap assert the following:

  • PYTHONPATH is unset

  • PYTHONHOME is unset

  • LD_PRELOAD is unset

  • PYTHONNOUSERSITE=1

  • VIRTUAL_ENV == UV_PROJECT_ENVIRONMENT

  • LD_LIBRARY_PATH only contains:

    • /nix/store/...

    • .runtime-state/host-libcuda

    • approved NVIDIA host-driver paths

If those invariants fail, the scripts should stop with an exact error instead of silently continuing with a polluted runtime.

Entry Points

Check prerequisites on a new host

Run this first on a new machine:

./scripts/nix/prereq-check.sh

It checks:

  • required commands such as nix, git, git-lfs, and uv

  • whether nix works with LD_LIBRARY_PATH cleared

  • free space on /nix and the workspace filesystem

  • supported NVIDIA driver file layout

  • required vendored dependencies such as third_party/curobo

  • obvious outer-shell pollution such as LD_LIBRARY_PATH, PYTHONPATH, or CONDA_PREFIX

Start the shell

direnv allow

or

nix --extra-experimental-features "nix-command flakes" develop

Bootstrap explicitly

./scripts/nix/bootstrap.sh

Use SIMPLE as a library

Inside the dev shell:

import gymnasium as gym
import simple.envs as _

env = gym.make("simple/FrankaTabletopGrasp-v0", sim_mode="isaac", headless=True)
obs, info = env.reset()
env.close()

Prefer this integration model over invoking the CLI from another Python application.

Run only the Python phase

./scripts/nix/bootstrap-python.sh

Run only the GPU phase

./scripts/nix/bootstrap-gpu.sh

Failure Model

The setup should fail early in these cases:

  • host NVIDIA runtime files are missing

  • runtime env variables are polluted by the outer shell

  • vendored dependencies are missing or uninitialized

  • GPU architecture cannot be detected automatically

  • /nix or the workspace filesystem is too full to realize the CUDA/Python closure reliably

That is deliberate. Silent fallback to host libraries is harder to debug and less reproducible than a hard assertion.

Why We Still Cannot Be Perfectly Host-Independent

Even with Nix managing almost all userspace, the following still come from the host:

  • running NVIDIA kernel driver

  • driver-provided user libraries matched to that kernel driver

  • Vulkan/EGL metadata installed by the driver package

So the reproducibility boundary is:

  • reproducible above the driver boundary

  • host-dependent at the driver boundary

That is the practical limit for GPU-heavy Linux development without fully containerizing or fully standardizing the host OS image.

Maintenance Rule

When changing the runtime:

  1. Put shared policy in scripts/nix/common.sh

  2. Keep flake.nix as orchestration only

  3. Put Nix-owned userspace in nix/runtime-base.nix

  4. Put host NVIDIA bridging in nix/runtime-host-gpu.nix

  5. Keep Python bootstrap separate from GPU bootstrap

  6. Keep the user-facing contract stable: prereq-check -> nix develop -> import simple

If a new fix requires copying shell logic into multiple files, that is usually a sign the fix belongs in common.sh instead.