At TwoSix, we needed to build a GCC cross toolchain for the obscure OpenRisc architecture in a repeatable way, and we did it with Nix. But before I delve into how, let’s talk about…
Why OpenRisc?
To answer the question of why, we first need a brief history lesson…
Little known fact, but there is actually OpenRisc silicon in the wild. In fact, before RISC-V went mainstream, OpenRisc was one of the go to license-free ISAs, perhaps along with Lattice Semiconductor’s ISA in the LatticeMico32.
A bit of an aside, making one’s own ISA is pretty straightforward. Adding GCC, LLVM, binutils, glibc, and Linux kernel support are much much trickier. Both OpenRisc and LatticeMico have C toolchain software support for GCC, LLVM, and the Linux Kernel.
Reasons to include OpenRisc or LatticeMico IP in hard silicon before RISC-V went mainstream probably went something like this:
At [insert company name here], we need a PMC(power management controller) in our chip but can’t really be bothered to poke ARM for yet another CPU license. Furthermore, since the PMC has only one job and is effectively software-isolated from the rest of the system, we can afford to use a non-standard ISA.
Back to the why of OpenRisc, which basically boils down to the fact that a particular project at TwoSix makes use of the very popular Allwinner A64 chip. The A64 has an OpenRisc PMC that can boot a software payload packaged into U-Boot. Building this payload requires the OpenRisc toolchain.
Challenges of Building A GCC Cross Compiler
Building a GCC cross compiler can be tricky for a number of reasons.
- The cross compiler has to be built in a number of stages in the right order.
- The cross compiler has to be built with the right version of GCC as future versions may have meaningful differences such as problematic Werror flags set by the Linux distro.
- The cross compiler has to be built with the right libc, binutils, and Linux kernel headers, and unfortunately, FSF doesn’t maintain a compatibility matrix.
Enter Nix
I’ve been using Nix to achieve repeatable builds as Nix allows me to pin down the exact environment in which a build occurs. Nix is better than Docker as Nix forces nearly purely functional builds. That is, Nix does its best to ensure that a build script will always produce identical outputs. In particular Nix does the following:
- Requires a “sources” stage for sources to be used when building a package. The sources can be fetched or can be local. Nix hashes the sources and only proceeds with the build if the sha256 source hashes are what the Nix recipe(derivation) said they would be.
- Performs builds with repeatable input state. Nix goes so far as to perform derivation builds in an offline chroot environment. As many sources of variation as possible are removed such included time(Nix fixes time at build to the Unix Epoch). The one thing Nix can’t really control for are race conditions in multithreaded builds that could potentially affect build outcomes. I suppose Nix could enforce only one core is allowed to perform builds(but that could be unusably slow). Furthermore, if the build relies on sources of entropy such as variations in memory access times, Nix can’t do much to enforce repeatability there.
- Stores successful builds in hashed paths. For example, say I want to spin up a Nix environment with gcc, bison, and verilator on my path, I would invoke
nix-shell --pure -p gcc bison verilator, and my path might look something like the following(path redacted to save space):
$ nix-shell --pure -p gcc bison verilator [nix-shell:~/git/uboot-2024-pine64]$ echo $PATH /nix/store/a11996pij79myrp43myn4zycdlzh9jsw-gcc-wrapper-13.3.0/bin:/nix/store/mcmyjzgqhfq6ycm0hz2482f454gd7ipc-gcc-13.3.0/bin:/nix/store/cv5w5wlyhfx1246dk4mdigj6x33w8jm2-glibc-2.39-52-bin/bin:/nix/store/y8yv3c6bnz15aqadfbkgs30xq88qwskm-verilator-5.022/bin:/nix/store/84yg60swk80b04apprb1432kir41bvzj-coreutils-9.5/bin
- Nix uses hashed stored builds as a cache for future builds. If Nix sees a recipe for a build that matches a hash of a build it has already stored/performed in that past, Nix will simple fetch that build store. Likewise, Nix can be pointed to build stores on other machines. The default Nix installation points to the upstream nixpkgs build store, but this is merely a matter of convenience. In principle, Nix can be pointed to any build store.
With that in mind, I can substantiate my claim of Nix being better than Docker – mainly because I’ve had issues in the past where old Ubuntu Docker containers responsible for building a python package might start to fail once upstream PyPi upgrades it certs. Nix mitigates this by fetching all the resources needed to build a given package ahead of time so that Nix builds always will work offline. This has some implications:
- Many upstream packages such as those written in packages have been patched to work with Nix offline build strategy such as Rust’s Cargo, Haskell’s Cabal, and even pip for Python(I think). This usually involves Nix computing the dependencies of a Rust crate for example, and then generating a nix script that will fetch and hash them.
- A Nix build doesn’t have to be dockerized – although it can be! Nix, once it has performed an initial build of a package will never again need internet access to perform subsequent builds provided that the Nix store cache has not been deleted(i.e. – you didn’t manually invoke nix garbage collection).
A GCC Cross Toolchain Buildscript in Nix
With this in mind, I feel confident in saying I have truly converged upon a repeatable GCC Cross toolchain builder nix derivation package. This Nix derivation should succeed where others have failed!
There are many GCC build scripts online that no longer work because the host GCC version on a modern distro is to different from the GCC that the script originally target – but no problem for Nix! The following derivation freezes the version of NixPkgs used, and by extension, every single dependency used to build gcc(and any other tool used in the derivation) forever.
Here is that glorious build script. If you want to just use the resulting OpenRisc toolchain, create both files below and then invoke nix-shell. After that, you should be able to invoke or1k-linux-gcc.
Lastly, this script should be able to be modified to support other architecture by changing the target variable’s triple. You may also have to fiddle around with mach_opts.
or1k-toolchain.nix
{ pkgs ? import (fetchTarball {
url = "https://github.com/NixOS/nixpkgs/archive/refs/tags/24.05.tar.gz";
}) {} }:
let
binutilsVersion = "2.36.1";
gccVersion = "11.2.0";
linuxKernelVersion = "6.6.30";
glibcVersion = "2.35";
target = "or1k-linux";
in
pkgs.stdenv.mkDerivation {
name = "or1k-toolchain";
version = "1.0";
srcs = [
( pkgs.fetchurl {
url = "http://ftpmirror.gnu.org/binutils/binutils-${binutilsVersion}.tar.gz";
sha256 = "sha256-5o7equtsqWh7bcuu3Rs3ZQa6rS1I3iaohfxatqy4Odo=";
})
( pkgs.fetchurl {
url = "http://ftpmirror.gnu.org/gcc/gcc-${gccVersion}/gcc-${gccVersion}.tar.gz";
sha256 = "sha256-8IN/G/gkSlzCO9lv9jZnEqeRz64B344lsTdpisom78E=";
})
( pkgs.fetchurl {
url = "https://www.kernel.org/pub/linux/kernel/v6.x/linux-${linuxKernelVersion}.tar.xz";
sha256 = "sha256-tmpbhjsPhmlEi3TKg71kGoVvFksplW5Tm7y1/e6rnMY=";
})
( pkgs.fetchurl {
url = "http://ftpmirror.gnu.org/glibc/glibc-${glibcVersion}.tar.xz";
sha256 = "sha256-USNzL2tnzNMZMF79OZlx1YWSEivMKmUYob0lEN0M9S4=";
})
];
buildInputs = [
pkgs.gcc
pkgs.wget
pkgs.which
pkgs.rsync
pkgs.gmp
pkgs.libmpc
pkgs.mpfr
pkgs.python3
pkgs.bison
pkgs.flex
];
# I believe the following prevents gcc from treating "-Werror=format-security"
# warnings as errors
hardeningDisable = [ "format" ];
sourceRoot = ".";
buildPhase = ''
echo $PWD
ls -lah .
# binutils
mkdir build-binutils
cd build-binutils
../binutils-${binutilsVersion}/configure \
--prefix=$out \
--with-sysroot=$out/sysroot \
--target=${target}
make -j$(nproc)
make install
cd ../linux-${linuxKernelVersion}
make ARCH=openrisc INSTALL_HDR_PATH=$out/sysroot/usr headers_install
cd ..
# gcc stage 1
mkdir build-gcc
cd build-gcc
../gcc-${gccVersion}/configure \
--target=${target} \
--prefix=$out \
--with-sysroot=$out/sysroot \
--enable-languages=c,c++ \
--disable-threads \
--disable-multilib \
--disable-libssp \
--disable-nls \
--disable-shared \
--with-gmp=${pkgs.gmp} \
--with-mpfr=${pkgs.mpfr} \
--with-mpc=${pkgs.libmpc}
make -j$(nproc) all-gcc
make install-gcc
cd ../
# build glibc headers
echo line 113
mkdir build-glibc
cd build-glibc
rm -rf *
echo "libc_cv_forced_unwind=yes" > config.cache
echo "libc_cv_c_cleanup=yes" >> config.cache
../glibc-${glibcVersion}/configure \
--host=${target} \
--prefix=/usr \
--with-headers=$out/sysroot/usr/include \
--config-cache \
--enable-add-ons=nptl \
--enable-kernel=${linuxKernelVersion}
make install_root=$out/sysroot install-headers
cd ..
# gcc stage 1.5
mkdir -p $out/${target}/include/gnu
touch $out/${target}/include/gnu/stubs.h
cd build-gcc
make -j$(nproc) all-target-libgcc
make install-target-libgcc
cd ..
# build glibc
export PATH=$out/bin:$PATH
echo line 138
cd build-glibc
rm -rf *
echo "libc_cv_forced_unwind=yes" > config.cache
echo "libc_cv_c_cleanup=yes" >> config.cache
mach_opts="-mcmov -mror -mrori -msfimm -mshftimm -msext -mhard-float"
export CC="${target}-gcc $mach_opts"
export AR="${target}-ar"
export RANLIB="${target}-ranlib"
export OBJCOPY="${target}-objcopy"
../glibc-${glibcVersion}/configure \
--host=${target} \
--prefix=/usr \
--libexecdir=/usr/lib/glibc \
--with-binutils=$out/bin \
--with-headers=$out/sysroot/usr/include \
--disable-werror \
--config-cache \
--enable-add-ons=nptl \
--enable-kernel=${linuxKernelVersion}
make -j$(nproc)
make install_root=$out/sysroot install
cd ..
# build glibc-final
cd build-gcc
rm -rf *
export CC="gcc"
export AR="ar"
export RANLIB="ranlib"
export OBJCOPY="objcopy"
../gcc-${gccVersion}/configure \
--with-gnu-ld \
--with-gnu-as \
--disable-nls \
--disable-libssp \
--with-multilib-list=mcmov,mhard-float \
--enable-languages=c,c++ \
--target=${target} \
--prefix=$out \
--with-sysroot=$out/sysroot
make -j$(nproc)
make install
cd ..
mkdir -p $out/${target}/lib
mkdir -p $out/sysroot/lib
rsync -a $out/${target}/lib/ $out/sysroot/lib
'';
meta = {
description = "Cross-compilation toolchain for OpenRISC architecture";
homepage = "https://openrisc.io";
license = pkgs.lib.licenses.gpl2;
maintainers = with pkgs.stdenv.lib.maintainers; [ ];
};
}
shell.nix
{ pkgs ? import (fetchTarball {
url = "https://github.com/NixOS/nixpkgs/archive/refs/tags/24.05.tar.gz";
}) {} }:
let
or1k-toolchain = pkgs.callPackage ./or1k-toolchain.nix {};
in
pkgs.mkShell {
buildInputs = [
or1k-toolchain
pkgs.which
];
}
Bridging the gap between the impossible and the practical with innovative technology solutions in cyber, data science, mobile, microelectronics and information operations and a full spectrum of products and capabilities to advance national security missions.