Revision as of 16:52, 2 December 2015

NIX
Description: User Level Purely Functional Package Manager
SHARCNET Package information: see NIX software page in web portal
Full list of SHARCNET supported software

Introduction

Nix is an atomic pure non-mutable package manager system that allows users to manage their own software environment. This environment is persistent and is shared across all clusters.

Users can build, install, upgrade, downgrade, and remove packages from their environment without root privileges and without affecting other users
Operations are atomic, they either succeed and create a new environment or fail leaving the previous environment in place. Previous environments can be switched back to at any point.

The default Nix package set includes a huge selection (over 10,000) of recent versions of many packages.

Version Selection

This section details how to enable your Nix environment and install and remove packages from it. See the #General Notes section for a more advanced discussion of how Nix works and how to create your own packages.

Enabling and disabling the Nix environment

The user's current Nix environment is enabled by loading the nix module. This creates some *.nix** files and sets some environment variables.

module load nix
ls -ld .nix*
env | fgrep -i nix

It is disabled by unloading the nix module. This unsets the environment variables but leaves the *.nix** files alone.

module unload nix
ls -ld .nix*
env | fgrep -i nix
module load nix

Installing, upgrading, uninstalling, and querying software

The nix environment is manipulated via the nix-env command. The --install (-i) option is used to install software

git --version
which git

nix-env --install git
git --version
which git

This takes a long time as it has to open many many files to enumerate all available software to locate the requested one by name. A faster way is to directly specify what to install by the attribute instead using the --attr (-a) option

nix-env --install --attr nixpkgs.subversion nixpkgs.mercurial

The --upgrade (-u) option upgrades installed software to the latest version. The --install (-i) option can also be used for this purpose.

nix-env --upgrade --attr nixpkgs.git

The --uninstall (-e) option uninstalls software. It can be done by name or path.

nix-env --uninstall $(which svn)

The contents of the environment is queried via the --query (-q) option.

nix-env --query

All possible packages can be queried via the --available (-a) option. This is quite slow again due to having to enumerate all available software. Adding --attr-path (-P) includes the faster attribute paths and --description includes the descriptions. It is a good idea to save this to a file for future reference.

nix-env --query --available --description --attr-path > nix-packages.txt
grep git nix-packages.txt | less -S

Environment transactions

Environments are non-mutable. All nix-env commands do not actually modify the current environment. Rather they create a new environment based on the existing environment and switch to it. This means all previous environments continue to exist and can be re-activated.

The --rollback option can be used to restore the previous environment.

nix-env --query
nix-env --rollback
nix-env --query

Previous generations of the environment can be displayed with the --list-generations option and any one can be switched to with the --switch-generation option.

nix-env --list-generations
nix-env --switch-generation 3
nix-env --query

Nix preserves any software this is accessible for any user environment. This means it is important to periodically delete older environment generations so Nix can remove the software associated with them. This is done with the --delete-generations options.

nix-env --delete-generations 30d

Environments are technically realizations of the Nix user-environment package.

Job Submission

Loading the Nix module before submitting a job makes the Nix environment available to the job. Note that these jobs will see the current Nix environment including any changes made after submission. Compiled binaries should not require the Nix module to be loaded to run..

Example Job

module load nix
sqsub ...

General Notes

This section details some of the internals of how Nix works and how to create your own packages. It is more advanced material not required for the basic usage detailed under the #Version Selection section above.

The Nix store

A Nix environment is just a collection of symlinks to all the packages that exist in that environment.

readlink $(which git)
readlink -f ~/.nix-profile

Everything in Nix exists as a path in the Nix store. Each path is distinguished with a hash of either its contents or everything that went into creating it to keep it separate from everything else. Paths are immutable once created. This means they can be freely shared.

Paths are created by the Nix build server when it realizes a Nix derivation. Nix derivations specify all the inputs to a jailed process that creates the contents of the store path.

Nix expressions and instantiation

Nix derivations are instantiated from Nix expressions, which are written in the Nix language. The nix-repl program can be used to interactively experiment with the Nix language.

nix-env --install --attr nix-repl
nix-repl

The following cabextract.nix Nix expression evaluates to a derivation fully specifying how to build the cabextract utility.

with import ~/.nix-defexpr/nixpkgs { };
stdenv.mkDerivation rec {
  name = "cabextract-1.6"

  src = fetchurl {
    url = "http://www.cabextract.org.uk/${name}.tar.gz";
    sha256 = "1ysmmz25fjghq7mxb2anyyvr1ljxqxzi4piwjhk0sdamcnsn3rnf";
  };

  meta = with stdenv.lib; {
    homepage = http://www.cabextract.org.uk/;
    description = "Free Software for extracting Microsoft cabinet files";
    platforms = platforms.all;
    license = licenses.gpl3;
    maintainers = with maintainers; [ pSub ];
  };
}

It is written using the stdenv.mkDerivation function from nixpkgs. This function creates a derivation that executes the standard unpack, patch, configure, build, check, install, fixup, check, and distribute steps on the package. It provides a series of hooks that can be used at each step to customize the process for non-standard packages.

The Nix expression is instantiate to a derivation using nix-instantiate. The leading ./ is required to distinguish that the arguments is file containing a Nix expression to be instantiated and not the name of a package in the default Nix expression to instantiate.

cabdrv=$(nix-instantiate ./cabextract.nix)
echo $cabdrv

Nix derivations and realization

The Nix derivation instantiated from the above Nix expression can be pretty-printed using the pp-aterm program.

nix-env --install --attr nixpkgs.strategoPackages.strategoxt
pp-aterm -i $cabdrv

Derive(
  [("out", "/home/nixbld/store/...-cabextract-1.6", "", "")]
, [ ("/home/nixbld/store/...-stdenv.drv", ["out"])
  , ("/home/nixbld/store/...-cabextract-1.6.tar.gz.drv", ["out"])
  , ("/home/nixbld/store/...-bash-4.3-p42.drv", ["out"])
  ]
, ["/home/nixbld/store/...-default-builder.sh"]
, "x86_64-linux"
, "/home/nixbld/store/...-bash-4.3-p42/bin/bash"
, ["-e", "...-default-builder.sh"]
, [ ("buildInputs", "")
  , ("builder", "/home/nixbld/store/...-bash-4.3-p42/bin/bash")
  , ("name", "cabextract-1.6")
  , ("nativeBuildInputs", "")
  , ("out", "/home/nixbld/store/...-cabextract-1.6")
  , ("propagatedBuildInputs", "")
  , ("propagatedNativeBuildInputs", "")
  , ("src", "/home/nixbld/store/...-cabextract-1.6.tar.gz")
  , ("stdenv", "/home/nixbld/store/...-stdenv")
  , ("system", "x86_64-linux")
  ]
)

The Nix build server realizes derivations by building the packages in an isolated environment. Each derivation specifies what comes out of the environment, what goes into it, the required machine architecture, what build program to execute in the environment, and what arguments and environment to pass to the program to be executed.

The nix-store program --realize option is used to signal the build server to realize the derivation.

cabpath=$(nix-store --realize $cabdrv)
echo $cabpath

The nix-env program will add the realized derivation to the users environment with the --install (-i) option. Technically it creates a new realization of the user-environment package that symlinks in the entire contents of the path path and switches to it.

nix-env --install $cabpath

The build log associated with a realization can be viewed with the --read-log (-l) option.

nix-store --read-log $cabdrv
nix-store --read-log $cabpath
nix-store --read-log $(which git)

The nix-store program also supports numerous other store related items such as computing the transitive closure of a package with the --query (-q) --requisites (-R) and exporting them via the --export option to a Nix archive for import into another store.

nix-store --query --requisites $cabpath
nix-store --export $(nix-store --query --requisites $(which git)) | gzip > git.nar.gz

Nix default expression

Nix has a default expression created from the contents of ~/.nix-defexpr. The nix-env --install operation normally works by selecting an attribute in this expression (fast) or matching against the name attribute in all the attributes in this expression (slow) to determine an expression to instantiate, realize, and then symlink into a new environment.

The nix module sets up a default ~/.nix-defexpr/nixpkgs that symlinks to a slightly modified version of the official Nix nixpkgs. This can be entirely replace this or augment. For example, the following ~/.nix-defexpr/mypkgs expression packages up the above cabextract.nix example.

args@{ ... }: with import ./nixpkgs args;
rec {
  cabextract = stdenv.mkDerivation rec {
    name = "cabextract-1.6";

    src = fetchurl {
      url = "http://www.cabextract.org.uk/${name}.tar.gz";
      sha256 = "1ysmmz25fjghq7mxb2anyyvr1ljxqxzi4piwjhk0sdamcnsn3rnf";
    };

    meta = with stdenv.lib; {
      homepage = http://www.cabextract.org.uk/;
      description = "Free Software for extracting Microsoft cabinet files";
      platforms = platforms.all;
      license = licenses.gpl3;
      maintainers = with maintainers; [ pSub ];
    };
  };
}

This makes it possible to install packages from mypkgs as easily as those from the official nixpkgs.

nix-env --install --attr mypkgs.cabextract

References

o Nix Package Manager
http://nixos.org/nix/

o Official Nix/Nixpkgs/NixOS
https://github.com/NixOS

o Nix on SHARCNET (slides by Tyson Whitehead)
https://www.sharcnet.ca/help/images/5/5f/Tyson_nix_2015.pdf

o Exploring a new approach to package management (youtube video by Tyson Whitehead)
https://www.youtube.com/watch?v=pQE9WTLAPHQ

Difference between revisions of "NIX"

Views