| NIX |
|---|
| Description: User Level Purely Functional Package Manager |
| SHARCNET Package information: see NIX software page in web portal |
| Full list of SHARCNET supported software |
Contents
Introduction
Nix is a package manager system that allows users to manage their own software environment. This environment is persistent and is shared across all clusters.
- Users can build, install, upgrade, downgrade, and remove packages from their environment without root privileges and without affecting other users.
- Operations either succeed and create a new environment or fail leaving the previous environment in place (operations are atomic).
- Previous environments can be switched back to at any point.
- Users can add their own packages and share them with other users.
The default Nix package set includes a huge selection (over 10,000) of recent versions of many packages.
Enabling and disabling the Nix environment
The user's current Nix environment is enabled by loading the nix module. This creates some .nix* files and sets some environment variables.
module load nix ls -ld .nix* env | fgrep -i nix
It is disabled by unloading the nix module. This unsets the environment variables but leaves the .nix* files alone.
module unload nix ls -ld .nix* env | fgrep -i nix module load nix
Completely reseting the Nix environment
Most operations can be undone with the --rollback option (i.e., nix-env --rollback or nix-channel --rollback). Sometimes it is useful to entirely reset nix though. This is done by unloading the module, erasing all user related nix files, and then reloading the module file.
module unload nix rm -fr ~/.nix-profile ~/.nix-defexpr ~/.nix-channels ~/.config/nixpkgs rm -fr /nix/var/nix/profiles/per-user/$USER /nix/var/nix/gcroots/per-user/$USER module load nix
Installing and remove packages
The nix-env command is used to setup your Nix environment.
What do I have installed and what can I install
Lets first see what we currently have installed.
nix-env --query
Now let's see what is available. We request the attribute paths (unambiguous way of specifying a package) and the descriptions too (cursor to the right to see them). This takes a bit of time as it visits a lot of small files. Especially over NFS it can be a good idea to pipe it to a file and then grep that in the future.
nix-env --query --available --attr-path --description
Installing packages
Let's say that we need a newer version of git than provided by default on our OS (e.g., the CentOS 6 one has issues with fetching over https). First lets check what our OS comes with.
git --version which git
Let's tell Nix to install its version in our environment.
nix-env --install --attr nixpkgs.git nix-env --query
Let's checkout what we have now (it may be necessary to tell bash to to forget remembered executable locations with hash -r so it notices the new one).
git --version which git
Removing packages
For completeness, lets add in the other usual version-control suspects.
nix-env --install --attr nixpkgs.subversion nixpkgs.mercurial nix-env --query
Actually, we probably don't really want subversion any more. Let's remove that.
nix-env --uninstall subversion nix-env --query
Environments
Nix keeps referring to user environments. Each time we install or remove packages we create a new environment based off of the previous environment.
Switching between previous environments
This means the previous environments still exist and we can switch back to them at any point. Let's say we changed our mind and want subversion back. It's trivial to restore the previous environment.
nix-env --rollback nix-env --query
Of course we may want to do more than just move to the previous environment. We can get a list of all our environments so far and then jump directly to whatever one we want. Let's undo the rollback.
nix-env --list-generations nix-env --switch-generation 4 nix-env --query
Operations are atomic
Due to the atomic property of Nix environments, we can't be left halfway through installing/updating packages. They either succeed and create us a new environment or leave us with the previous one intact.
Let's go back to the start when we just had Nix itself and install the one true GNU distributed version control system tla. Don't let it complete though. Hit it with CTRL+c partway through.
nix-env --switch-generation 1 nix-env --install --attr nixpkgs.tla
CTRL+c
Nothing bad happens. The operation didn't complete so it has no effect on the environment whatsoever.
nix-env --query nix-env --list-generations
Nix only does things once
The install and remove commands take the current environment and create a new environment with the changes. This works regardless of which environment we are currently in. Let's create a new environment from our original environment by just adding git and mercurial.
nix-env --list-generations nix-env --install nixpkgs.git nixpkgs.mercurial nix-env --list-generations
Notice how much much faster it was to install git and mercurial the second time? That is because the software already existed in the local Nix store from the previous installs so Nix just reused it.
Garbage collection
Nix periodically goes through and removes any software not accessible from any existing environments. This means we have to explicitly delete environments we don't want anymore so Nix is able to reclaim the space. We can delete specific environments or any sufficiently old.
nix-env --delete-generations 30d
Using channels to obtain Nix expressions
Nix packages are Nix expressions that specify how to build something. The default source for of these expression for nix-env is ~/.nix-defexp and ~/.nix-defexp/channels. Nix channels provide a way to populate this later directory with existing expressions from repositories on the internet.
Subscribing to a channel
By default we are subscribed to the SHARCNET version of the latest NixOS release under the name nixpkg. Let us add the older NixOS 15.09 release too under the name nixpkgs_old and then download the latest versions of all NixOS 15.09 the expressions.
nix-channel --list nix-channel --add file:///nix/channels/sharcnet-15.09 nixpkgs_old nix-channel --list nix-channel --update nixpkgs_old
Installing packages from a channel
Now searching our available packages we see that we can install software using either the nixpkgs or nixpkgs_old expressions.
nix-env --query --available --attr-path --description git
Let's replace our git built and installed from the unstable (nixpkgs) expression with one built and installed from the stable expression (note that git is the default version of git which is gitMinimal).
nix-env --install --attr nixpkgs_old.git nix-env --query git --version
Remove/Read Package & Channel Example
1) Lets clear the slate and uninstall atop and nixpkgs for this example ... If you have never used nix before on the system this step is unnecessary then goto step 2) [roberpj@gra-login1:~] module load nix [roberpj@gra-login1:~] nix-env --query atop-2.3.0 nix-index-0.1.0 [roberpj@gra-login1:~] nix-env --uninstall atop uninstalling 'atop-2.3.0' [roberpj@gra-login1:~] nix-channel --remove nixpkgs uninstalling 'nixpkgs' [roberpj@gra-login1:~] nix-env --query nix-index-0.1.0 2) Lets now install nixpkgs, which is a ONE TIME required operation for new nix users ... [roberpj@gra-login1:~] module load nix [roberpj@gra-login1:~] nix-channel --add file:///nix/channels/ccpkgs/latest nixpkgs [roberpj@gra-login1:~] nix-channel --update unpacking channels... [roberpj@gra-login1:~] module unload nix 3) Now install atop, or any other nix package for that matter ... [roberpj@gra-login1:~] module load nix [roberpj@gra-login1:~] nix-env --install --attr nixpkgs.atop replacing old 'atop-2.3.0' installing 'atop-2.3.0' [roberpj@gra-login1:~] which atop ~/.nix-profile/bin/atop
4) Next time we login, to use atop simply load the nix module ! [roberpj@gra-login1:~] module load nix [roberpj@gra-login1:~] which atop ~/.nix-profile/bin/atop
What about dependency conflicts
There are no issues with having a mix of packages installed from different sources even if they have conflicting dependencies. Nix puts each package in a separate directory under /nix/store versioned by the hash of the its build instructions. Binaries are linked with rpaths to ensure they always find the versions they need.
ldd $(readlink ($which git))
Switching between environments continue to work as before.
nix-env --rollback git --version nix-env --list-generations nix-env --switch-generation 5 git --version
Switching between previous updates
The channel updates (updating the list of expressions we can build and install from) are also atomic and versioned. This ensures we never find ourselves stuck due to accidentally updating to something broken.
nix-channel --rollback nix-env --query --available --attr-path git nix-channel --rollback 2
Development with Nix (e.g., Python, R, etc.)
Several languages have their own repository of packages and associated infrastructure (e.g., PyPI and pip). Nix builds on these to automatically handle the external dependencies (e.g., C libraries) and makes it easy to work simultaneously with different projects requiring different versions of internal packages.
Wrapper scripts for setting paths
Nix has expressions for many of these that generate wrapper scripts that set path environment variables to bring a specific set of the internal packages into scope and then run the approriate program.
The following Nix expression defines top-level myPython, myR, and myHaskell attributes that use the appropriate package-specific expressions to create wrappers for a selection of internal packages (for details about the Nix langauge see the Nix Expression Language.
with import <nixpkgs> { };
rec {
myPython = python3.buildEnv.override rec {
extraLibs = with python3Packages; [ numpy scipy ];
};
myR = rWrapper.override rec {
packages = with rPackages; [ rgeos rgdal ];
};
myHaskell = haskellPackages.ghcWithPackages (ghcpkgs:
with ghcpkgs; [ pipes lens cabal-install ]
);
}
Saving it as ~/.nix-defexpr/mypkgs.nix makes these available for installation with nix-env.
nix-env --install --attr mypkgs.myPython cat $(which python3) python3 import scipy
CTRL+d
Development environments
A Nix expression evaluates to a specification for setting up a (build) environment and then doing a build in it. Frequently we want to do something very similar: setup a (development) environment and then do development in it.
The nix-shell command bridges the gap between these two. It uses Nix to sets up a non-rooted (build) environment with the dependencies we specify and then laucnhes a (development) shell in it instead of a build process.
mkdir devel cd devel
Say we need Clang and Python with the NumPy and SciPy packages. Instead of installing these into our global environment we can create a Nix expression that specifies these as build (development) inputs
with import <nixpkgs> { };
stdenv.mkDerivation {
name = "my-env";
buildInputs = [
( python3.buildEnv.override rec {
extraLibs = with python3Packages; [ numpy scipy ];
} )
clang
];
}
save it in default.nix and then run nix-shell.
nix-shell
Now we are in a build (development) environment with all the dependencies we specified in the appropriate PATHs.
cat $(which python3) clang --version python3 import scipy
CTRL+d
To return to our standard environment we just exit the shell. This is extreamily nice if we work on many projects with conflicting dependencies.
exit cd ..
One-off environments
Quite frequently we need a one-off development environment with a few packages. Say CLang and git. Rather than have to write the following boilerplate default.nix file
with import <nixpkgs> { };
stdenv.mkDerivation {
name = "my-env";
buildInputs = [
clang
git
];
}
we can just get nix-shell to do it for us.
nix-shell --packages clang git
CTRL+d
Submitting Jobs
Loading the Nix module before submitting a job makes the Nix environment available to the job. Note that these jobs will see the current Nix environment including any changes made after submission. Compiled binaries should not require the Nix module to be loaded to run.
module load nix sqsub ...
Internals
This section details some of the internals of how Nix works and how to create your own packages. It is more advanced material not required for the basic usage.
The Nix store
A Nix environment is just a collection of symlinks to all the packages that exist in that environment.
readlink $(which git) readlink -f ~/.nix-profile
Everything in Nix exists as a path in the Nix store. Each path is distinguished with a hash of either its contents or everything that went into creating it to keep it separate from everything else. Paths are immutable once created. This means they can be freely shared.
Paths are created by the Nix build server when it realizes a Nix derivation. Nix derivations specify all the inputs to a jailed process that creates the contents of the store path.
Nix expressions and instantiation
Nix derivations are instantiated from Nix expressions, which are written in the Nix language. The nix-repl program can be used to interactively experiment with the Nix language.
nix-env --install --attr nix-repl nix-repl
The following cabextract.nix Nix expression evaluates to a derivation fully specifying how to build the cabextract utility.
with import ~/.nix-defexpr/nixpkgs { };
stdenv.mkDerivation rec {
name = "cabextract-1.6"
src = fetchurl {
url = "http://www.cabextract.org.uk/${name}.tar.gz";
sha256 = "1ysmmz25fjghq7mxb2anyyvr1ljxqxzi4piwjhk0sdamcnsn3rnf";
};
meta = with stdenv.lib; {
homepage = http://www.cabextract.org.uk/;
description = "Free Software for extracting Microsoft cabinet files";
platforms = platforms.all;
license = licenses.gpl3;
maintainers = with maintainers; [ pSub ];
};
}
It is written using the stdenv.mkDerivation function from nixpkgs. This function creates a derivation that executes the standard unpack, patch, configure, build, check, install, fixup, check, and distribute steps on the package. It provides a series of hooks that can be used at each step to customize the process for non-standard packages. For full details see the nixpkgs manual.
The Nix expression is instantiate to a derivation using nix-instantiate. The leading ./ is required to distinguish that the arguments is file containing a Nix expression to be instantiated and not the name of a package in the default Nix expression to instantiate.
cabdrv=$(nix-instantiate ./cabextract.nix) echo $cabdrv
Nix derivations and realization
The Nix derivation instantiated from the above Nix expression can be pretty-printed using the pp-aterm program.
nix-env --install --attr nixpkgs.strategoPackages.strategoxt pp-aterm -i $cabdrv
Derive(
[("out", "/home/nixbld/store/...-cabextract-1.6", "", "")]
, [ ("/home/nixbld/store/...-stdenv.drv", ["out"])
, ("/home/nixbld/store/...-cabextract-1.6.tar.gz.drv", ["out"])
, ("/home/nixbld/store/...-bash-4.3-p42.drv", ["out"])
]
, ["/home/nixbld/store/...-default-builder.sh"]
, "x86_64-linux"
, "/home/nixbld/store/...-bash-4.3-p42/bin/bash"
, ["-e", "...-default-builder.sh"]
, [ ("buildInputs", "")
, ("builder", "/home/nixbld/store/...-bash-4.3-p42/bin/bash")
, ("name", "cabextract-1.6")
, ("nativeBuildInputs", "")
, ("out", "/home/nixbld/store/...-cabextract-1.6")
, ("propagatedBuildInputs", "")
, ("propagatedNativeBuildInputs", "")
, ("src", "/home/nixbld/store/...-cabextract-1.6.tar.gz")
, ("stdenv", "/home/nixbld/store/...-stdenv")
, ("system", "x86_64-linux")
]
)
The Nix build server realizes derivations by building the packages in an isolated environment. Each derivation specifies what comes out of the environment, what goes into it, the required machine architecture, what build program to execute in the environment, and what arguments and environment to pass to the program to be executed.
The nix-store program --realize option is used to signal the build server to realize the derivation.
cabpath=$(nix-store --realize $cabdrv) echo $cabpath
The nix-env program will add the realized derivation to the users environment with the --install (-i) option. Technically it creates a new realization of the user-environment package that symlinks in the entire contents of the path path and switches to it.
nix-env --install $cabpath
The build log associated with a realization can be viewed with the --read-log (-l) option.
nix-store --read-log $cabdrv nix-store --read-log $cabpath nix-store --read-log $(which git)
The nix-store program also supports numerous other store related items such as computing the transitive closure of a package with the --query (-q) --requisites (-R) and exporting them via the --export option to a Nix archive for import into another store.
nix-store --query --requisites $cabpath nix-store --export $(nix-store --query --requisites $(which git)) | gzip > git.nar.gz
Nix default expression
Nix has a default expression created from the contents of ~/.nix-defexpr. The nix-env --install operation normally works by selecting an attribute in this expression (fast) or matching against the name attribute in all the attributes in this expression (slow) to determine an expression to instantiate, realize, and then symlink into a new environment.
The nix-channel command works by building new Nix expression packages (containing all the Nix expressions for the subscribed channels) and symlinking ~/.nix-defexprs/channels to then. Adding additional expressions to ~/.nix-defexpr makes them available for use with nix-env as well. For example, the following ~/.nix-defexpr/mypkgs expression packages up the above cabextract.nix example.
args@{ ... }: with import ./nixpkgs args;
rec {
cabextract = stdenv.mkDerivation rec {
name = "cabextract-1.6";
src = fetchurl {
url = "http://www.cabextract.org.uk/${name}.tar.gz";
sha256 = "1ysmmz25fjghq7mxb2anyyvr1ljxqxzi4piwjhk0sdamcnsn3rnf";
};
meta = with stdenv.lib; {
homepage = http://www.cabextract.org.uk/;
description = "Free Software for extracting Microsoft cabinet files";
platforms = platforms.all;
license = licenses.gpl3;
maintainers = with maintainers; [ pSub ];
};
};
}
This makes it possible to install packages from mypkgs as easily as those from the official nixpkgs.
nix-env --install --attr mypkgs.cabextract
References
o Nix Package Manager
http://nixos.org/nix/
o Official Nix/Nixpkgs/NixOS
https://github.com/NixOS
o Nix on SHARCNET (slides by Tyson Whitehead)
https://www.sharcnet.ca/help/images/5/5f/Tyson_nix_2015.pdf
o Exploring a new approach to package management (youtube video by Tyson Whitehead)
https://www.youtube.com/watch?v=pQE9WTLAPHQ