commit	c566f69cc01c37cfebfd727a8a966c6828b2e762	[log] [tgz]
author	George Burgess IV <gbiv@google.com>	Mon Jan 09 14:40:49 2023 -0700
committer	Chromeos LUCI <chromeos-scoped@luci-project-accounts.iam.gserviceaccount.com>	Sat Jan 14 13:44:23 2023 +0000
tree	95f73d24d90a59d7f2118727e0649c4627468e5a
parent	b5ef3dff57c9f94558b55d300eb0dc949ed808b6 [diff]

tree: 95f73d24d90a59d7f2118727e0649c4627468e5a

README.md

Vendored Rust Libraries

This repository contains vendored Rust libraries for the system bluetooth project, and for ChromeOS' dev-rust/third-party-crates-src ebuild. It is generated by running cargo vendor to store all our dependencies.

Please see the individuals listed in the OWNERS file if you'd like to learn more about this repo.

Updating packages

In order to update or add any package, follow these steps:

Find the project in projects/ corresponding to the first-party package you'd like to update. If it does not exist, please see the "Adding a first-party package," section.
Modify its Cargo.toml to add, remove or upgrade packages.
Run python vendor.py
- This runs cargo vendor first, which updates Cargo.lock and puts downloaded crates into the vendor directory
- It applies any patches in the patches directory. It also regenerates checksums for packages that were modified.
- It removes OWNER files from packages that have it (interferes with our own OWNERS management) and regenerates checksums
- It checks that all packages have a supported license and lists the set of licenses used by the crates.
- If --license-map=<filename> is given, it will dump a json file which is a dictionary with the crate names as keys and another dictionary with the license and license_file as keys.
Verify that no patches need to be updated. Patches in patches/ can either be named after a crate (in which case, they apply to all versions of their corresponding crate), or can have a version in their name, in which case they're meant to apply specifically to a given version of a crate.
If any licenses are unsupported, do the following:
- Check if the package is actually included in the build. cargo vendor seems to also pick up dependencies for unused configs (i.e. windows). You will need to make sure these packages are stripped by cargo vendor.
- Check if the license file exists in the crate's repository. Sometimes the crate authors just forget to list it in Cargo.toml. In this case, add a new patch to apply the license to the crate and send a patch upstream to include the license in the crate as well.
- Check if the license is permissive and ok with ChromeOS. If you don't know how to do this, reach out to the OWNERS of this repo.
  - Yes: Update the vendor.py script with the new license and also update dev-rust/third-party-crates-src with the new license.
  - No: Do not use this package. Contact the OWNERS of this repo for next steps.

NOTE: If your patch changes over 30 files, please see the Splitting Patches section below.

Adding patches

When it is necessary to patch a package due to incompatibility, you can create a patch targeting the specific package and store it in patches/. For any given ${crate} at ${version}, if patches/${crate}-${version} exists, the patches from that directory are applied to the crate. If such a directory does not exist, patches/${crate} is checked. Similarly, if this exists, patches are applied; otherwise, the crate is left unpatched.

If ./vendor.py complains about a specific directory in patches/ not having a corresponding vendor/ directory, the most likely fixes are:

the crate is no longer required, and the patches should be deleted.
the crate has been upgraded, and the patches that were previously applied should be evaluated for whether they are still relevant.

Patches can come in two forms. Files with names ending in .patch are always applied with patch -p1 to the vendor directory. Executable files that do not have names ending in patch will be executed in the vendor directory which they should apply to. All other files are ignored.

Testing updates

Updates to this repo will be captured by the CQ. To directly test changes, either build the net-wireless/floss package, or run cros_workon --board=${BOARD} dev-rust/third-party-crates-src and build packages + run tests for your board of choice.

Adding a first-party package

The projects/ subdirectory contains a set of Cargo.toml files that roughly correspond to what exists in the ChromeOS tree. These exist only to provide dependency information with which we can create our vendor/ directory, so everything is removed except for author information, dependencies, and (if necessary) a minimal set of features/annotations. Dependency sections outside of what would build for ChromeOS, like [target.'cfg(windows)'.dependencies], are also removed.

Once your Cargo.toml seems correct and you're ready to test, run projects/populate-workspace.py to add it to the workspace.

Admittedly, it's sort of awkward to have two Cargo.tomls for each first-party project. It may be worth trying to consolidate this in the future, though our future bazel migration potentially influences what the 'ideal' setup here is.

NOTE: If your patch changes over 30 files, please see the Splitting Patches section below.

What is "a minimal set of features/annotations"?

cargo vendor will trim the vendor directory based on the features you require of your dependencies, but not their optionality. For instance, given:

[dependencies]
tokio = { version = "1", optional = true }

cargo vendor will simply vendor tokio as though the optional = true isn't present. That said, it's important to note that cargo vendor will try to trim optional dependencies of crates which Cargo.toml files depend upon. For instance, given:

[dependencies]
tokio = { version = "1", features = [] }

cargo vendor selects a much smaller set of crates to vendor than it would if features = [] were replaced with features = ["full"]. Hence, Cargo.tomls should be sure to either specify all of the features they require explicitly in a dependency's features = [] block, or keep a crate-level [features] block which makes the set of potential features of dependencies we use visible to cargo vendor. For example, both of the following two Cargo.toml bodies are OK:

[dependencies]
tokio = { version = "1", features = ["full"] }

[dependencies]
tokio = { version = "1", features = [] }

[features]
tokio-full = [ "tokio/full" ]

tl;dr: if your crate has optional dependencies, feel free to drop the optional = annotation. If your crate depends on features of other crates, please do not remove those.

My crate's name conflicts with another first party's

The crate names in these Cargo.tomls aren't really relevant or used; please add some context that seems meaningful to your package's name, or do that to the conflicting package.

Splitting Patches

Due to the nature of rust_crates, patches can get massive (tens-hundreds of thousands of lines of diff over thousands of files). For ease of review, it's requested that large changes are split into a few logically independent pieces:

All changes outside of vendor/, crab/, and projects/Cargo.lock.
Changes to crab/.
Changes to vendor/ and projects/Cargo.lock.

The following patch stack roughly serves as an example of this:

These patches should all be landed at the same time. The split is meant to help increase reviewability. If you're unfamiliar with Gerrit's Relation chain feature, you can make a patch stack like the above by making multiple commits in git, and running repo upload as usual.

If you have one large patch that you'd like to split out into three, git checkout may be helpful to you:

## Assuming you currently have the full change you want to commit (e.g.,
## CRAB changes, vendor/ changes, and all other changes all in one commit)
## checked out, and `git status` says your repo is clean:
$ all_changes=$(git rev-parse HEAD)
$ git checkout HEAD~ -- vendor crab projects/Cargo.lock
$ git commit --amend --no-edit
$ git checkout "${all_changes}" -- crab
$ git commit crab    # This commits crab/ changes.
$ git checkout "${all_changes}" -- vendor projects/Cargo.lock
$ git commit vendor  # This commits vendor changes.

## NOTE: if any part of this fails, you can always get back to your original
## state by running `git reset --hard ${all_changes}`.

To ensure your splitting did the right thing, you can run git diff ${all_changes}. If no changes are printed and git status is clean, you successfully split your CL.

In order to make changes to old CLs, git rebase -i and/or git commit --fixup combined with git rebase -i --autosquash might be helpful to you. Please see git's manual for more info.

FAQ

Why is `vendor.py` complaining about CRAB audits?

Please see the CRAB readme.

How do I make my changes go live in `dev-rust/third-party-crates-src`?

dev-rust/third-party-crates-src is a cros_workon package. If you'd like to hack on it for a board, you can use cros-workon-${BOARD} start dev-rust/third-party-crates-src. If you'd like changes to apply to the host, use cros-workon --host start dev-rust/third-party-crates-src instead. After running one of these, any emerges of dev-rust/third-party-crates-src should output that the 9999 version of this package is being emerged, and your local changes should apply.

As an example, for local development on dependencies for the host, the "add a crate and make the change live" cycle looks like:

$ cros-workon --host start dev-rust/third-party-crates-src
$ while ! rust_crates_does_what_i_want; do
   $EDITOR projects/foo/Cargo.toml && \
   ./vendor.py && \
   sudo emerge dev-rust/third-party-crates-src
 done