commit	e8d1457716520d8bc6a2a51c83c503bac303404d	[log] [tgz]
author	Allen Webb <allenwebb@google.com>	Thu Dec 08 15:47:48 2022 +0000
committer	Chromeos LUCI <chromeos-scoped@luci-project-accounts.iam.gserviceaccount.com>	Wed Dec 14 16:39:09 2022 +0000
tree	07d20642cf1e9829c92b91dbc1fd0c44df686c6b
parent	c8e97608689962f25e525cd3d143b0224409e9f3 [diff]

tree: 07d20642cf1e9829c92b91dbc1fd0c44df686c6b

README.md

Vendored Rust Libraries

This repository contains vendored Rust libraries for the system bluetooth project, and for ChromeOS' dev-rust/third-party-crates-src ebuild. It is generated by running cargo vendor to store all our dependencies.

Please see the individuals listed in the OWNERS file if you'd like to learn more about this repo.

Updating packages

In order to update or add any package, follow these steps:

Find the project in projects/ corresponding to the first-party package you'd like to update. If it does not exist, please see the "Adding a first-party package," section.
Modify its Cargo.toml to add, remove or upgrade packages.
Run python vendor.py
- This runs cargo vendor first, which updates Cargo.lock and puts downloaded crates into the vendor directory
- It applies any patches in the patches directory. It also regenerates checksums for packages that were modified.
- It removes OWNER files from packages that have it (interferes with our own OWNERS management) and regenerates checksums
- It checks that all packages have a supported license and lists the set of licenses used by the crates.
- If --license-map=<filename> is given, it will dump a json file which is a dictionary with the crate names as keys and another dictionary with the license and license_file as keys.
Verify that no patches need to be updated. Patches in patches/ can either be named after a crate (in which case, they apply to all versions of their corresponding crate), or can have a version in their name, in which case they're meant to apply specifically to a given version of a crate.
If any licenses are unsupported, do the following:
- Check if the package is actually included in the build. cargo vendor seems to also pick up dependencies for unused configs (i.e. windows). You will need to make sure these packages are stripped by cargo vendor.
- Check if the license file exists in the crate's repository. Sometimes the crate authors just forget to list it in Cargo.toml. In this case, add a new patch to apply the license to the crate and send a patch upstream to include the license in the crate as well.
- Check if the license is permissive and ok with ChromeOS. If you don't know how to do this, reach out to the OWNERS of this repo.
  - Yes: Update the vendor.py script with the new license and also update dev-rust/third-party-crates-src with the new license.
  - No: Do not use this package. Contact the OWNERS of this repo for next steps.

NOTE: If your patch changes over 30 files, please see the Splitting Patches section below.

Adding patches

When it is necessary to patch a package due to incompatibility, you can create a patch targeting the specific package and store it in patches/. For any given ${crate} at ${version}, if patches/${crate}-${version} exists, the patches from that directory are applied to the crate. If such a directory does not exist, patches/${crate} is checked. Similarly, if this exists, patches are applied; otherwise, the crate is left unpatched.

If ./vendor.py complains about a specific directory in patches/ not having a corresponding vendor/ directory, the most likely fixes are:

the crate is no longer required, and the patches should be deleted.
the crate has been upgraded, and the patches that were previously applied should be evaluated for whether they are still relevant.

Patches can come in two forms. Files with names ending in .patch are always applied with patch -p1 to the vendor directory. Executable files that do not have names ending in patch will be executed in the vendor directory which they should apply to. All other files are ignored.

Testing updates

Updates to this repo will be captured by the CQ. To directly test changes, either build the net-wireless/floss package, or run cros_workon --board=${BOARD} dev-rust/third-party-crates-src and build packages + run tests for your board of choice.

Adding a first-party package

The projects/ subdirectory contains a set of Cargo.toml files that roughly correspond to what exists in the ChromeOS tree. These exist only to provide dependency information with which we can create our vendor/ directory, so everything is removed except for author information, dependencies, and (if necessary) a minimal set of features/annotations. Dependency sections outside of what would build for ChromeOS, like [target.'cfg(windows)'.dependencies], are also removed.

Once your Cargo.toml seems correct and you're ready to test, run projects/populate-workspace.py to add it to the workspace.

Admittedly, it's sort of awkward to have two Cargo.tomls for each first-party project. It may be worth trying to consolidate this in the future, though our future bazel migration potentially influences what the 'ideal' setup here is.

NOTE: If your patch changes over 30 files, please see the Splitting Patches section below.

What is "a minimal set of features/annotations"?

cargo vendor will trim the vendor directory based on the features you require of your dependencies, but not their optionality. For instance, given:

[dependencies]
tokio = { version = "1", optional = true }

cargo vendor will simply vendor tokio as though the optional = true isn't present. That said, it's important to note that cargo vendor will try to trim optional dependencies of crates which Cargo.toml files depend upon. For instance, given:

[dependencies]
tokio = { version = "1", features = [] }

cargo vendor selects a much smaller set of crates to vendor than it would if features = [] were replaced with features = ["full"]. Hence, Cargo.tomls should be sure to either specify all of the features they require explicitly in a dependency's features = [] block, or keep a crate-level [features] block which makes the set of potential features of dependencies we use visible to cargo vendor. For example, both of the following two Cargo.toml bodies are OK:

[dependencies]
tokio = { version = "1", features = ["full"] }

[dependencies]
tokio = { version = "1", features = [] }

[features]
tokio-full = [ "tokio/full" ]

tl;dr: if your crate has optional dependencies, feel free to drop the optional = annotation. If your crate depends on features of other crates, please do not remove those.

My crate's name conflicts with another first party's

The crate names in these Cargo.tomls aren't really relevant or used; please add some context that seems meaningful to your package's name, or do that to the conflicting package.

Splitting Patches

Due to the nature of rust_crates, patches can get massive (tens-hundreds of thousands of lines of diff over thousands of files). For ease of review, it's requested that large changes are split into a few logically independent pieces:

All changes outside of vendor/, crab/, and projects/Cargo.lock.
Changes to crab/.
Changes to vendor/ and projects/Cargo.lock.

The following patch stack roughly serves as an example of this:

These patches should all be landed at the same time. The split is meant to help increase reviewability. If you're unfamiliar with Gerrit's Relation chain feature, you can make a patch stack like the above by making multiple commits in git, and running repo upload as usual.

If you have one large patch that you'd like to split out into three, git checkout may be helpful to you:

## Assuming you currently have the full change you want to commit (e.g.,
## CRAB changes, vendor/ changes, and all other changes all in one commit)
## checked out, and `git status` says your repo is clean:
$ all_changes=$(git rev-parse HEAD)
$ git checkout HEAD~ -- vendor crab projects/Cargo.lock
$ git commit --amend --no-edit
$ git checkout "${all_changes}" -- crab
$ git commit crab    # This commits crab/ changes.
$ git checkout "${all_changes}" -- vendor projects/Cargo.lock
$ git commit vendor  # This commits vendor changes.

## NOTE: if any part of this fails, you can always get back to your original
## state by running `git reset --hard ${all_changes}`.

To ensure your splitting did the right thing, you can run git diff ${all_changes}. If no changes are printed and git status is clean, you successfully split your CL.

In order to make changes to old CLs, git rebase -i and/or git commit --fixup combined with git rebase -i --autosquash might be helpful to you. Please see git's manual for more info.

FAQ

Why is `vendor.py` complaining about CRAB audits?

Please see the CRAB readme.

How do I make my changes go live in `dev-rust/third-party-crates-src`?

dev-rust/third-party-crates-src is a cros_workon package. You can either use cros-workon-${BOARD} start dev-rust/third-party-crates-src or ~/chromiumos/chromite/scripts/cros_uprev --force --overlay-type public --packages dev-rust/third-party-crates-src. If you don't do these, emerge-${BOARD} dev-rust/third-party-crates-src will internally check out an old version of rust_crates, which will effectively discard your changes.

If you choose to run cros_uprev, please do not commit those changes. Annealing and the CQ will handle that for you.

Migration-specific FAQ

FAQ that's only relevant while we're migrating from dev-rust to third-party-crates-src.

FIXME(b/240953811): Remove this section and all subsections once the migration is done.

`third-party-crates-src` is dying because a package doesn't exist. Why?

We keep an allowlist in dev-rust/third-party-crates-src that you'll have to update. Please run ~/chromiumos/src/third_party/chromiumos-overlay/dev-rust/third-party-crates-src/files/write_allowlisted_crate_versions.py, and upload the changes that makes. Make sure to add a Cq-Depend from rust_crates to the CL you uploaded to chromiumos-overlay, and vice-versa.

How do I make my newly-added package go live ASAP?

tl;dr: contact gbiv@.

THe current migration strategy for third-party-crates-src is for it to iteratively replace the leaf packages of dev-rust. Concretely, imagine the following depgraph (where A -> B means "A depends on B"):

my-cool-package
-> tokio-1.0.0
  -> anyhow-1.0.0
    -> libc-0.2.0
      -> compiler_builtins-0.1.1

Even if third-party-crates-src has tokio-1.0.0, it won't export tokio-1.0.0 until anyhow-1.0.0 is exported, it won't export anyhow-1.0.0 until libc-0.2.0 is exported, and it won't export libc-0.2.0 until compiler_builtins-0.1.1 is exported.

This leaves-in migration is necessary to minimize the chances of us accidentally messing up portage DEPENDs, for making migration patches readable, and for avoiding race conditions (b/247596883#comment3).

If we need to export a non-leaf crate, the crates which it depends on may also have to be exported. Further, all of these need to be inspected to ensure they won't introduce race conditions, among other things. Rather than building up a process and automation for this just to tear it all down in a few weeks, gbiv@ has volunteered to do it all as folks request it. Yay. :)