Writing Rust ebuilds

This is a short reference, intended to be read alongside Basic guide to write Gentoo Ebuilds and the cargo.eclass documentation.

cargo.eclass

cargo.eclass is an eclass for utilizing Rust's own package manager, Cargo. Documentation for the eclass is available in the relevant section of the Development Guide and in the cargo.eclass(5) man page provided by the app-doc/eclass-manpages package.

Cargo is very convenient for software developers. However, it comes with some caveats for maintainers.

Rust programs are usually statically bound. This makes runtime dependencies easier to figure out compared to C programs, but build dependencies are harder to get right. The good thing is that upstream needs to figure that out for packagers. Packagers only need to translate what upstream already figured out into the cargo.eclass way of handling dependency programs.

Rust's dependency mechanism

Rust dependencies come in the form of 'crates'. Crates can be libraries or runnable programs. Like the term 'package', 'crate' is not defined strictly, so basically every Rust program that is provided in binary or source form can be considered a crate. In the context of packaging, crates are often dependency libraries of a package. Rust developers upload their crates along with documentation to crates.io. Other Rust developers can use the uploaded crates in their program by stating its name and version in a configuration file for Rust's build system, Cargo. If a dependency is only declared using the crate name, Cargo assumes it should download that dependency from crates.io.

However, developers can also declare dependencies with additional information, like a URL to a Git Repository and a Git tag, to get crates from elsewhere.

Making a versioned ebuild

Gentoo has its own program to make ebuilds out of Rust projects automatically: app-portage/pycargoebuild.

To use pycargoebuild, obtain the repository of the Rust software being packaged (e.g. by cloning the repository), change to the directory containing the sources, and generate an ebuild by running:

If everything goes well, there will now be an ebuild in the repository, which can then be moved into an ebuild repository like ::gentoo or ::guru. However, note that if Cargo.toml doesn't have the fields homepage and description, the HOMEPAGE and DESCRIPTION variables in the ebuild will need to be added, and if the package is published on crates.io, it can be added to the CRATES variable (probably as ${PN}@${PV}), and if it lives as an archive elsewhere, it should be added to SRC_URI. For cases where dependencies are not all on crates.io, see the common problems section for more information.

Licenses

Automated ebuilds will have LICENSE assigned, but the value of this variable is not guaranteed to be correct. To generate a list of a Rust project's licenses, run:

in the Rust repository.

Packagers must be aware that cargo license will give the licenses in SPDX format, which Gentoo does not always use. So some effort needs to be put into 'translating' them.

Because the code of the dependencies is compiled into the finished binary (statically linked), all licenses of every crate used in a package must be stated in the LICENSE variable.

Common problems with dependencies in versioned ebuilds

crates.io dependencies

If all dependencies are fetched from crates.io, the automatically generated ebuilds will often work from scratch. The crates.io dependencies are added to the CRATES variable in the ebuild, combining their name and version number with the '@' symbol, e.g.:

CRATES="
 addr2line@0.21.0
 adler@1.0.2
 adler32@1.2.0
 ...
"

and also add ${CARGO_CRATE_URIS} to the SRC_URI variable, e.g.:

SRC_URI="${CARGO_CRATE_URIS}"

These dependencies will automatically be fetched and unpacked into the right place via cargo_src_unpack, from cargo.eclass.

GitHub/GitLab etc. dependencies

Sometimes upstream developers are not happy with the version of a crate on crates.io, or a crate is not available there and must instead be obtained from a git repository. In these situations, such a dependency is specified in Cargo.toml like this:

tree-sitter-haxe={git="https://github.com/vantreeseba/tree-sitter-haxe",version="0.2.2",optional=true}

The above specifies that release 0.2.2 from the tree-sitter-haxe GitHub repository should be checked out.

The user can't clone a git repository without git-r3.eclass, but the user can simulate this in the ebuild. In this example, the user would add the lines:

declare -A GIT_CRATES=(
 [tree-sitter-haxe]="https://github.com/vantreeseba/tree-sitter-haxe;32f6bda9b568ae47c89678096de9b4d0cbd450b8"
)

to the ebuild.

The commit hash is obtained by browsing the files of that release (v0.2.2) on GitHub and copying it from the URL. It can also often be found in the Cargo.toml and Cargo.lock files of the upstream repository, by grepping the repository for the crate's name.

If the Cargo.toml of a Rust program has a line like:

tree-sitter-c-sharp = { git = "https://github.com/tree-sitter/tree-sitter-c-sharp", branch = "master" }

in its dependencies, it wants to fetch the latest commit on the master branch of that GitHub repository. This essentially means packagers must make a live ebuild, rather than a versioned ebuild.

Usually the cargo.eclass looks for Cargo.toml in the folder $WORKDIR/$crate_name-$commit-hash. Thus, in the above example, the folder would be $WORKDIR/tree-sitter-haxe-32f6bda9b568ae47c89678096de9b4d0cbd450b8.

However, some crates have Cargo.toml in a different place, or have a different name in Cargo.toml as their GitHub repository name. In such cases, the path to Cargo.toml can be defined using:

declare -A GIT_CRATES=(
 [tree-sitter-haxe]="https://github.com/vantreeseba/tree-sitter-haxe;32f6bda9b568ae47c89678096de9b4d0cbd450b8;tree-sitter-haxe-%commit%/mypath/to/cargotoml"
)

where %commit% is automatically replaced with the commit hash specified.

If the last part of that line is not provided, the cargo.eclass composes that path out of the crate name (the part in []) and the commit hash.

The path is a relative path that starts from $WORKDIR. The part in [] usually needs to be the name of the package in the Cargo.toml of that package.

Patching or updating crates

As the vast majority of Rust dependencies are in crates with fixed versions for a release, inevitably there will be a vulnerable (or otherwise outdated) crate that eventually needs to be patched. There are four ways to patch crates in a Gentoo ebuild.

Making a new release

The first (and most complex) is making a new release and dependency tarball (or just a new release that works with updated CRATES). This is relatively straightforward, but requires the most development effort. This approach is most useful where there are a number of crates that require updates, different versions are required, and replacements are non-trivial. This process is documented elsewhere, though the high level process involves using cargo update to update the Cargo metadata files Cargo.{lock,toml} to include updated but compatible crates.

Manually patching crates

The second is manually patching the crate(s) in question within the ebuild environment. We don't often do this in Gentoo as there is most often an updated crate on crates.io, however it is sometimes unavoidable. When patching crates it is important to remember that the .cargo-checksum.json file must be updated to include the new checksum for each file that the patch changes. It is recommended that this approach only be used where required, and only for patches that cannot be fetched as an updated compatible crate from crates.io.

See Adelie's Rust packaging for a minimal example. The same approach can be used to patch a crate in CRATES, if required, but it may be better to just fix it upstream and ask for a new release.

Using cargo.eclass

Finally there are the two cargo.eclass -assisted methods:

Substituting a compatible crate

As the eclass unpacks dependencies into an offline repository and configures cargo to use this repository in place of crates.io, we can substitute compatible crates into CRATES then run cargo update to update the package against these dependencies. To do this, replace the crate in CRATES and then call the cargo_update_crates helper function in src_prepare. This calls cargo update with appropriate options to update the Cargo.{lock,toml} files at build-time.

CRATES="
 	adler@1.0.2
-	autocfg@1.1.0
+	autocfg@1.2.0
 	bindgen@0.65.1
 	bit_field@0.10.1
 	bitflags@1.3.2

. . .

# This whole phase is defined as cmake_src_prepare otherwise overrides it;
# it was not previously required.
+src_prepare() {
+	cargo_update_crates
+	cmake_src_prepare
+ }

Overriding all crates with a particular name

paths = [...] overrides: An array of paths to directories containing a Cargo.toml. cargo will read the Cargo.toml and override any crate with the same name in the dependency graph of the (cargo) package. This does not respect version constraints or the lock file. The eclass will automatically configure this for crates in CRATE_PATHS_OVERRIDE; crates in this variable are added to CRATES by the eclass, though it will not hurt to have them in both.

Test the package extensively to ensure that no new bugs have been introduced. This method is primarily intended to support the replacement of vendored crates; consider whether it is more appropriate to substitute a new crate in CRATES.

# Requried to build against openssl-3.*
CRATE_PATHS_OVERRIDE="
 	openssl@0.10.35
	openssl-sys@0.9.65
"
. . .
src_unpack() {
	verify-sig_src_unpack
	cargo_src_unpack
	# Vendored sources here override crates-io sources (repo) from cargo eclass.
	sed -i '/\[source.crates-io\]/,+2d' "${ECARGO_HOME}"/config.toml
}

The src_unpack step is only necessary where the package in question overrides crates.io in its cargo config.

Writing live ebuilds

First create the skeleton ebuild as described above. Then change the version in the ebuild's file name to 9999.

In the ebuild itself:

• Remove the unnecessary CRATES variable.
• Remove the unnecessary SRC_URI variable; sources are typically fetched via git-r3.eclass or similar. git-r3 should come after cargo on the inherit line.
• Create a src_unpack() function, containing cargo_live_src_unpack:

src_unpack() {
 git-r3_src_unpack
 cargo_live_src_unpack
}

This will fetch the source from git and then the needed dependency crates using Cargo.

Problems with live ebuilds

"can't update a git repository in the offline mode"

When a Rust project uses unpinned dependencies in its Cargo.toml, e.g.:

tree-sitter-c-sharp={git="https://github.com/tree-sitter/tree-sitter-c-sharp",branch="master"}

Cargo will always want to check if that dependency is up-to-date in later stages of the ebuild. This will result in the error "can't update a git repository in the offline mode".

To fix this, add cargo_src_configure --frozen to src_configure(). This will stop Cargo from checking whether what it just fetched is up-to-date.

USE flags

USE flags for Rust are usually added via cargo.eclass:

src_configure(){
localmyfeatures=(
barfeature
$(usevfoo)
)
cargo_src_configure
}

Switching features off

Often Rust programs do not allow switching individual default features off. So in order to only enable certain default features, disable all default features with cargo_src_configure --no-default-features, then enable the desired features via myfeatures, as described above.

Unbundling C libraries

Start with constructing a dependency tree:

There's a naming convention[1] for crates linking with native libraries to have a -sys suffix. The user's job as a maintainer is to find out which features pull them and put corresponding native libraries into package dependencies.

However, most crates use vendored C libraries by default (see bug #709568), which is discouraged by Gentoo policies.

pkg-config crate

If this crate is pulled, the user need to add virtual/pkgconfig to ebuild's BDEPEND and explicitly allow cross-compilation:

exportPKG_CONFIG_ALLOW_CROSS=1

Common -sys crates

exportJEMALLOC_OVERRIDE="${ESYSROOT}/usr/$(get_libdir)/libjemalloc.so"

exportOPENSSL_NO_VENDOR=1

exportSNAPPY_LIB_DIR="${ESYSROOT}/usr/$(get_libdir)"
exportROCKSDB_LIB_DIR="${ESYSROOT}/usr/$(get_libdir)"
exportROCKSDB_INCLUDE_DIR="${ESYSROOT}/usr/include"

Other -sys crates

First, inspect crate's Cargo.toml for features that force static linking. If they are enabled by any revdeps, the user has run out of luck.

Then, inspect crate's build.rs script for environment variables that control build flow. Set them in ebuild's src_configure() or global scope.

Using a vendor tarball like in Go ebuilds

The go-module.eclass and the cargo.eclass are very similar regarding the functionality of loading statically linked dependencies. In a Go ebuild a packager defines the variable EGO_SUM with a list of dependencies from upstream, in a Rust ebuild the CRATES variable has the same functionality. The eclasses then proceed to download those dependencies and put them in the correct places in the working directory.

While it is an ongoing and controversial discussion whether the EGO_SUM-functionality should be deprecated or not in the go-module.eclass, the cargo.eclass is built with the CRATES variable in mind. The go.eclass offers the possibility to use a vendor-tarball instead of the EGO_SUM-functionality. That is a tar archive that contains all the statically linked dependencies. While Cargo also offers the possibility to create such a vendor-tarball as easily as Go, the cargo.eclass does not offer to use such a vendor-tarball without doing some extra steps.

Using a vendor-tarball instead of using the CRATES variable has some benefits. For example, the user don't need to recreate the list of dependencies (e.g. by using pycargoebuild in the upstream repo) and copy it into the new ebuild when bumping the version of that ebuild. If the vendor tarball is hosted right, bumping a Rust-ebuild is as easy as renaming the ebuilds filename with the new version.

Automatically creating tarballs with gentoo-crate-dist

Vendor tarballs can be automatically created using gentoo-crate-dist, which uses GitHub actions to create tarballs for each tag of a given repository. To add packages to the official gentoo-crate-dist, users can submit a bug here, which should follow the format:

Repository URL: The repository URL for the desired package. Official locations should be preferred over mirrors.
Initial tags needed: The first tag a tarball is needed for, usually the latest.
Special pycargoebuild arguments needed to get the tarball: Sometimes additional directories are needed. Leave as N/A if unsure.

Alternatively, users can create their own version of gentoo-crate-dist using projg2/crate-dist-mirror-action.

Manually creating a vendor tarball

Creating a vendor tarball can be done by running cargo vendor in the upstream repository. It will create a vendor folder, which can be put into a tar archive with XZ_OPT='-T0 -9' tar -acf vendor.tar.xz vendor.

Note that current versions of cargo no longer generate a vendor-config.toml file and instead output the equivalent to stdout. You may still require a vendor-config.toml file however (see below). You can create one yourself like so: cargo vendor > vendor-config.toml && mv vendor-config.toml vendor/

Using a vendor tarball in an ebuild

Because the cargo.eclass is somewhat built with the CRATES variable in mind, it will complain when this variable is not set when the eclass in inherited. To work around this, the user can define the variable as CRATES=" ". The cargo.eclass looks for dependencies in $ECARGO_VENDOR directory which defaults to $CARGO_HOME/gentoo. $CARGO_HOME itself is set to $WORKDIR/cargo_home. A packager can either

1. change default variable by setting $ECARGO_VENDOR to other location before src_unpack() or
2. unpack the vendor-tarball into default $ECARGO_VENDOR directory, (by not calling default or cargo_src_unpack in src_unpack() and writing src_unpack() with own instructions.) or
3. let portage do its default thing and let it unpack the vendor tarball into $WORKDIR, then do ln -s "${WORKDIR}/<extracted directory>/"* "${CARGO_HOME}/gentoo/".

In the case that a packages uses crates from git instead of crates.io cargo needs additional configuration. Luckily the vendor tarball has that configuration already included in a file called vendor-config.toml. This file expects the directory with the vendored sources in the working directory of the package itself. So when the vendor tarball is unpacked in $WORKDIR/vendor link it to the packages workdir like this for example: ln -s "${WORKDIR}/vendor/" "${WORKDIR}/lapce-${PV}/vendor" || die (Set the path accordingly) The contents of vendor-config.toml collide with some settings that the cargo-eclass creates during cargo_gen_config in ${ECARGO_HOME}/config.toml. Those settings need to be deleted:

sed -i "${ECARGO_HOME}/config.toml" -e '/source.crates-io/d' || die
sed -i "${ECARGO_HOME}/config.toml" -e '/replace-with = "gentoo"/d' || die
sed -i "${ECARGO_HOME}/config.toml" -e '/local-registry = "\/nonexistent"/d' || die

Then the contents of vendor-config.toml can be appended to ${ECARGO_CONFIG}/config.toml: cat "${WORKDIR}/vendor/vendor-config.toml" >> "${ECARGO_HOME}/config.toml" || die

See also

• Basic guide to write Gentoo Ebuilds — getting started writing ebuilds, to harness the power of Portage, to install and manage even more software.
• User:Schievel/autocreate_rust_sources

External resources

• Documentation of cargo.eclass