Packaging Gradle software with Nix

Published: at 09:12 AM

Packaging Gradle software with Nix

Some time ago, I thought it would be neat to use Nix to build Android apps. If an Android app could be built with a Nix Flake, then, in theory, a whole F-Droid-like repository of reproducible apps could be made. F-Droid currently seems to struggle with building apps reproducibly: they document all the ways in which .APKs may differ from build to build. You can find F-Droid’s documentation on reproducible builds here.

Given this idea, I set out to create a flake which could build an open-source Android app, hoping that the end result would be deterministic.

After some research (i.e. procrastinating on university work by googling around instead),I found Robotnix, which uses Nix to build Android images. It turns out that as a side-product, they already build some APKs from source, while other apps are just fetched from pre-built binaries. Unfortunately, simply trying to build a Robotnix app from source fails, since they’re using an ancient version of Nixpkgs, and are riddled with hacks made to force Gradle to work under Nix.

Why is this? Even their F-Droid package was built from source at some point, but the project switched to a pre-built version.

The problem: Gradle

Most (if not all?) Android apps are built using Gradle. Nix (more precisely, Nixpkgs) doesn’t have any standard way of building Gradle software: some packages use fixed-output derivation hacks, others use perl magic together with patching. The most common approach, however, as adopted by Robotnix, is to use gradle2nix to build a nix derivation of all Gradle dependencies of software package. Sadly, gradle2nix is currently abandonware and refuses to work against current Nixpkgs channels.

My first thought was to get gradle2nix working again, but this experiment ended quickly. Even if I had gotten it working, it’s currently a whole Kotlin app that would require continuous maintenance. Plus, I’ve never used Kotlin before. A different approach was in order.

Thankfully, my previous project (read: another college procrastination) was packaging maven-mvnd using Nix (available as an output in my personal flake, if you’re interested!), which gave me some ideas on how to approach this. The current way to build Maven software with Nix, as adopted by Nixpkgs by means of buildMavenPackage, is described by Fareed Zakaria in his blog.

Essentially, buildMavenPackage creates a fixed-output derivation where maven is invoked to evaluate all project dependencies and store them in a local maven repository, so it can be used when building that same project.

(^footnote 1: in practice, buildMavenPackage defaults to building the whole project inside the fixed-output derivation for network access, since Maven is known for not being able to fetch all dependencies a task (e.g, build) will need without actually running that task, despite having a dependency:go-offline command that should do exactly that. Some Gradle projects may suffer from the same problem.)

The work: Gradle cache internals

Given this idea, I picked the smallest app I could find in robotnix, which was Auditor, and set out to make it build without gradle2nix and patching hacks.

My first task was to understand how Gradle works and how it fetches dependencies. I left some useful Gradle documentation which put me in the right path in the references below.

In the end, my approach abuses Gradle’s new dependency verification feature, which is a modest attempt at protecting a project’s dependencies against tampering and security risks by checksumming and optionally signing them. In the end, this mechanism stores each dependency’s hash and signature inside a file in the project’s repository, to be committed for use by later Gradle invocations and other members. It also forces Gradle to download all these dependencies to the local cache (in a best-effort approach), which is exactly what I was looking for.

I say it’s a best-effort approach because, as the docs warn, Gradle may discover different dependencies depending on the tasks that it runs. This means that, in a worst-case scenario, a whole project may have to be built inside a fixed-output derivation to force Gradle to discover all build-time dependencies.

After many, many long hours of trial-and-error, I managed to build a derivation which contained all dependencies Auditor needed:


  # This is what makes everything work:
  # --write-verification-metadata forces discovered dependencies to be fetched to cache
  GRADLE_ARGS = "--console plain --no-daemon --write-verification-metadata sha512";

  dependencies = stdenv.mkDerivation {
    name = "gradle-home-dependencies";
    nativeBuildInputs = [ jdk gradle ];
    inherit src ANDROID_HOME;
    dontFixup = true;

    buildPhase = ''
      GRADLE_USER_HOME=$(pwd)/.gradle gradle ${GRADLE_ARGS} help lint
    '';
    installPhase = ''
      # See here for a mapping of gradle version and the respective cache paths:
      # https://docs.gradle.org/8.4/userguide/dependency_resolution.html#sub:ephemeral-ci-cache

      mkdir -p $out/caches/modules-2
      cp -a .gradle/caches/modules-2/. $out/caches/modules-2/

      # Delete extra files to ensure a stable hash
      find $out -type f -regex '.+\\(\\.lastUpdated\\|resolver-status\\.properties\\|_remote\\.repositories\\|\\.lock\\)' -delete
      find $out -type f \( -name "*.log" -o -name "*.lock" -o -name "gc.properties" \) -delete
    '';
    outputHashAlgo = "sha256";
    outputHashMode = "recursive";
    # outputHash = lib.fakeHash;
    outputHash = "sha256-NYol439do1IDk4Qpq1sNXNZOV1mn+8VDwgr+uu1Cu/4=";
  };

Note: even though I don’t like it, I had to pass the Android SDK to this derivation because the lint task depends on it, and linting dependencies weren’t being fetched without actually executing the task. I couldn’t for the life of me disable linting in the actual build derivation, because there’s too many linting tasks to disable manually.

After getting a Gradle cache inside a derivation, now it’s only a matter of instructing a simple gradle build derivation to use it:

stdenv.mkDerivation rec {
  pname = "auditor";
  inherit version;
  name = "${pname}-${version}";

  inherit src dependencies ANDROID_HOME;
  nativeBuildInputs = [ jdk gradle ];
  buildPhase = ''
    mkdir .gradle
    # Copy the whole gradle cache to a writeable path, since gradle wants to write more files into the $GRADLE_USER_HOME folder.
    cp -R --no-preserve=all ${dependencies}/. .gradle/
    # Note: Nix Wiki makes use of $GRADLE_OPTS for setting additional gradle arguments, but this environment variable has since been deprecated:
    # https://nixos.wiki/wiki/Android#gradlew
    GRADLE_USER_HOME=$(pwd)/.gradle gradle build --offline ${GRADLE_ARGS}
  '';

  # Ommitted the installPhase for simplicity: Basically, copy the resulting .APKs to $out and discard everything else.
  # See the end of the post for a link to the whole code.
}

Let’s try building:

❯ nix build .#auditor
(...)
❯ tree result/
result
├── debug
│  ├── app-debug.apk
│  └── output-metadata.json
├── play
│  ├── app-play-unsigned.apk
│  └── output-metadata.json
└── release
   ├── app-release-unsigned.apk
   └── output-metadata.json

It works! :tada: We can now build fully working APKs, using Nix, from a fixed collection of inputs.

My first instinct was to verify if these APKs were reproducible:

# Rebuilds force nix to verify if the existing outputs matches those of this new build
❯  nix build .#auditor --keep-failed --print-build-logs --rebuild
(...)
note: keeping build directory '/tmp/nix-build-auditor-78.drv-16'
error: derivation '/nix/store/<A>.drv' may not be deterministic: output '/nix/store/<A>' differs from '/nix/store/<A>.check'

Well, bummer. Fortunately, F-Droid’s reproducible builds documentation referred me to use diffoscope to easily see the differences between two APKs:


❯ cd /tmp/nix-build-auditor-78.drv-16/source/app/build/outputs/
❯ nix run nixpkgs#diffoscope -- ./ ~/repos/robotnix/result/
(...)
│ │   --- ./debug/app-debug.apk
│ ├── +++ ~/repos/robotnix/result/debug/app-debug.apk
│ │┄ No differences before APK Signing Block; not running 'apktool'.
│ │ ├── APK Signing Block
│ │ ├── /nix/store/5zhi93ygap11a90cqkdhmr4wwklw91hi-apksigner-33.0.1/bin/apksigner verify --verbose --print-certs
        [Ommited the output because it was too long;]
        [Basically, the signature hashes don't match.]

│ │ │ @@ -2,15 +2,15 @@
│ │ │  Verified using v1 scheme (JAR signing): false
│ │ │  Verified using v2 scheme (APK Signature Scheme v2): true
│ │ │  Verified using v3 scheme (APK Signature Scheme v3): false
│ │ │  Verified using v4 scheme (APK Signature Scheme v4): false
│ │ │  Verified for SourceStamp: false
│ │ │  Number of signers: 1
│ │ │  Signer #1 certificate DN: C=US, O=Android, CN=Android Debug
│ │ │ +Signer #1 certificate SHA-256 digest: c4289753ffac6caee2a53cd3712eb3b7f9d8ba0378ca79c3fd708aa47537a422
│ │ │ +Signer #1 certificate SHA-1 digest: 63fc7cbafd2d750ecc1ae2ae96da4754451e7b66
│ │ │ +Signer #1 certificate MD5 digest: 7efc21f3e2d3a15dfcb123962580f03a
│ │ │ -Signer #1 certificate SHA-256 digest: 3f90d452ce2050d3e1f73cabc891bd937fd87aada0c6f873da7fd2b426b0ebc7
│ │ │ -Signer #1 certificate SHA-1 digest: d86e63cd8bc977893b0553d8d802464704499fa6
│ │ │ -Signer #1 certificate MD5 digest: c2afbe6ced0ef6ca5257186e8f62af1c
│ │ │  Signer #1 key algorithm: RSA
│ │ │  Signer #1 key size (bits): 2048
│ │ │ +Signer #1 public key SHA-256 digest: 64f2c641f823348de85a503bea8962ec89ead8f290f0c6a790690a6aff675be3
│ │ │ +Signer #1 public key SHA-1 digest: 9469bd44566d738a2670118acac6642447202307
│ │ │ +Signer #1 public key MD5 digest: 6b2deed4d158215acc44496845538ca9
│ │ │ -Signer #1 public key SHA-256 digest: 81b4109c1007746b615b359da6db5f367c8bf88e3102927b840ca45acae5fbac
│ │ │ -Signer #1 public key SHA-1 digest: 76eec0d91a342fd0a9ff7b68ccdd630402c070f7
│ │ │ -Signer #1 public key MD5 digest: 581c5f3c67bfd44185f2ad72ada47b89

It seems app-debug.apk is the culprit here. Apparently, this apk is signed with a randomly-generated key each time, so the output is never deterministic due to this.

However, since diffoscope doesn’t complain about anything else, this means the other two .APKs don’t differ! Reproducible builds were achieved! ^[2]

insert great success gif here

footnote 2: Well, at least on the same build machine, which is good enough for me.

Conclusion

In the end, I enjoyed this project because it was simple to implement in theory, but I had to go down a whole rabbit-hole of issues to get it working in practice. This forced me (in a good way!) to learn more of how the Nix, Gradle and Android ecosystems work. Some things, like using Maven and Gradle in Nix are scarcely documented, which forced me to read a bunch of source files, issues, PRs and others to understand how it all works.

I hope to generalize this approach into a general-purpose function, in the style of buildMavenPackage. I also hope to eventually propose this approach to Gradle projects to Nixpkgs, so the whole community benefits from a more low-maintenance and universal approach to packaging Gradle projects.

You can find the final default.nix file, as well as all the changes I made to Robotnix at the time of this post in this branch.

References:

Special thanks to my friends that helped me by proof-reading, providing feedback and kept incentivizing me to start writing things.