IzzyOnDroid Reproducible Builds Transparency Log (rbtlog)

Here you find the recipes and logs from the rbtlog builder run by Izzy. It covers mostly apps hosted at the IzzyOnDroid repo.

👁 Contributor Covenant

Below is its original Readme:

👁 CI
👁 podman
👁 docker
👁 update log
👁 update recipes
👁 AGPLv3+

Reproducible Builds Transparency Log (rbtlog)

rbtlog is a Reproducible Builds transparency log for Android APKs. Its git repository contains:

• Scripts forming a rebuilder framework.
• Recipes to build various apps.
• Rebuild Logs forming a transparency log of reproduction attempts.
• GitHub Actions workflows to automate everything.

Announcing Android Reproducible Builds at IzzyOnDroid with rbtlog

We are pleased to announce "Reproducible Builds, special client support and more in our repo": a collaboration between various independent interoperable projects: the IzzyOnDroid team, 3rd-party clients Droid-ify & Neo Store, and rbtlog (part of my collection of tools for Android Reproducible Builds) to bring Reproducible Builds to IzzyOnDroid and the wider Android ecosystem.

- Fay

Security

The current build setup only supports building inside a container (using podman or docker), which does not provide the isolation of a full VM (though hopefully libvirt/debvm support will be added eventually). There are some safeguards against trivial container breakouts, rootless podman is supported and recommended, and the build runs with as few privileges as possible inside the container, but there are no guarantees. GitHub Actions uses ephemeral VMs, which does reduce the risk somewhat when using that to build.

Commits should always be reviewed and signed using the key specified in about.json (which the build infrastructure should absolutely not have access to). Anyone running their own rebuilder instance should evaluate the risks involved. We do not recommend building untrusted apps without further safeguards.

Scripts

The scripts in scripts/ provide the rebuilder component.

build.py

Builds unsigned APKs from apps' recipes and compares them to the signed APKs published by the upstream developer(s) using apksigcopier.

For each specified <appid>:<tag>, reads recipes/<appid>.yml and builds the requested tag from its recipe (using the specified backend, e.g. podman or docker) and produces JSON output on stdout; with --verbose also produces status messages and a build log on stderr.

Local Builds

With --local, the specification can be APPID:TAG:[COMMIT]:[APK|none] (instead of just APPID:TAG) to build a specific commit instead of a tag (but still using the recipe from the specified tag) and/or compare to a local APK. Or to just build an APK from a specific commit without any Reproducible Builds comparison.

The APK is either none (no comparison), empty (use APK from recipe), an http(s) URL, or a local file path.

update-log.py

Updates the transparency log by rebuilding all tags for the specified apps' recipes that are not yet part of the log.

For each specified /path/to/<appid>.yml (e.g. recipes/*.yml), makes a list of <appid>:<tag> pairs not already in logs/<appid>.json, runs build.py to build them, and adds the resulting output to logs/<appid>.json.

update-recipes.py

Updates the apps' recipes by checking upstreams' forges for new latest releases.

For each specified /path/to/<appid>.yml (e.g. recipes/*.yml), checks the relevant forge (e.g. GitHub) for the latest release tag (and APK URL) and adds a new entry in the recipe (unless that tag already has an entry).

See update-hashes.py below for information on how --update-hashes works and when update-hashes.py is called.

update-hashes.py

NB: this script will also update cmdline_tools, node.js LTS, and/or reproducible-apk-tools downloads to the latest version in versions.json.

Generally, update-recipes.py does not modify the build: steps, or depend on their contents in any way: it simply copies the recipe from the previous tag when adding a new one to the YAML recipe.

Unfortunately, some recipes require modifications to build: steps, which is what update-hashes.py tries to automate by downloading the upstream APK and making required modifications to the build: steps. It currently supports 2 specific things: embedded commit hashes and .dex/.prof/.profm hashes for flaky builds.

The script can be called directly, but is mostly intended to be used via update-recipes.py, which calls it automatically: with --update-hashes it is called for all recipes that don't opt-out by using the no-update-hashes label, otherwise it is called for all recipes that opt-in by using the update-hashes label.

Embedded commit hashes

Android Gradle Plugin automatically embeds the commit hash since version 8.3:

Unfortunately, an APK built before committing and tagging the release will thus have the wrong embedded commit hash. In order to make Reproducible Builds work in these cases where only the embedded commit hash differs, a simple git reset --soft $EMBEDDED_COMMIT_HASH can be used to embed the correct hash whilst still building from the exact same source code the tag points to.

To automate this, update-hashes.py will check for an embedded commit hash in the upstream APK and prepend such a git reset --soft line to the build: steps when the embedded hash doesn't match the tag's. Any existing git reset --soft or git checkout lines are always removed.

Flaky builds: .dex/.prof/.profm hashes

Sometimes builds are "flaky": different clean builds produce different APKs, only one of which matches upstream's. In these cases, common practice is rebuilding multiple times until the hash of the intermediate .dex or .prof or .profm file that is different between builds matches the one from upstream's APK.

For build recipes that look like the examples below, update-hashes.py will get the SHA-1 hash of the file from the upstream APK and update the DEX_SHA1=, PROF_SHA1=, or PROFM_SHA1= line in the build: steps:

Version updates: versions.json

The following will be updated to use the latest release as specified in versions.json (which can itself be updated using update-versions.py):

The cmdline_tools: in provisioning:; a git clone -b vX.Y.Z https://github.com/obfusk/reproducible-apk-tools.git in build:; and a node.js LTS download in build: that looks like:

You can opt out of these updates with the no-update-cmdline-tools, no-update-nodejs-lts, and/or no-update-repro-apk labels, respectively.

update-versions.py (& versions.json)

Updates versions.json, which looks like:

make-index.py

Processes each specified /path/to/<appid>.json rebuild log file and produces a JSON index on stdout. This index maps each upstream APK's SHA-256 checksum to a summary of its rebuild results.

NB: work in progress; output format may change.

NB: ideally there are only rebuild results for the tag corresponding to the version code/name of the APK, but this cannot be guaranteed as it may have been attached to another tag (by mistake).

provision-root.sh & provision.sh

These scripts are used by build.py to provision the build environment (e.g. in a podman container): installing required packages and the Android SDK, cloning the app's repository, etc.

Recipes

The YAML recipes in recipes/ provide the (re)build instructions for individual apps. For example, the build recipe for Catima looks like this (with all but the last version elided and example labels added):

The updates: specification can be releases[:...] (either bare releases or with a regex, e.g. releases:v.*\d$), tags: followed by a regex (e.g. tags:([\d.]+)), manual, disabled, or checkonly: followed by releases[:...] or tags:... (to check but not update).

For tags: You can use not only the full matched tag ($$TAG$$) but also any capturing group ($$TAG:1$$ for the first group) as a placeholder in apk_url.

For releases with multiple per-ABI APKs, use multiple build recipes under apks:.

Rebuild Logs

The JSON rebuild logs in the logs/ directory of this git repository form a transparency log of reproduction attempts.

NB: this directory, index.json, and about.json are only present on the (default) log branch, which is otherwise identical to the master branch.

For example, the rebuild log for Catima looks like this:

GitHub Actions workflows

ci.yml, podman.yml, docker.yml

CI for the rebuilder itself (code linting etc.).

update-log.yml

Automatically runs scripts/update-log.py -v podman recipes/*.yml every day and creates a pull request with the changes. The pull request is reviewed and signed before being merged into the log branch.

update-recipes.yml

Automatically runs scripts/update-recipes.py -v recipes/*.yml every day and creates a pull request with the changes. The pull request is reviewed and signed before being merged into the master and log branches.

JSON Schemas

There are JSON Schemas to validate the YAML recipes, JSON logs, and index.json in schemas/.

Installing

Everything but the dependencies is contained in the git repository.

$ git clone https://github.com/obfusk/rbtlog.git # main repository @ github.com
$ git clone https://gitlab.com/obfusk/rbtlog.git # mirror @ gitlab.com
$ git clone https://codeberg.org/obfusk/rbtlog.git # mirror @ codeberg.org

Dependencies

Python >= 3.9 + several libraries (jsonschema, requests, ruamel.yaml) + apksigcopier + reproducible-apk-tools + a backend like podman or docker.

Debian/Ubuntu

$ apt install podman python3-pip # can also use docker.io instead of podman
$ pip install -r requirements.txt # may need/want to use a venv

Docker security

$ cat /etc/docker/daemon.json
{
 "userns-remap": "default"
}

License

👁 AGPLv3+