If you just want to use Linux, you should read the Running Custom Containers Under Chrome OS doc. This doc is all about how the crostini is made, not how to use it. :)
Note: this diagram is not exhaustive, but covers the major services and how they interact.
Googlers: update this image at go/termina-rpc
| Name | Affects | Repo | ebuild |
|---|---|---|---|
| 9s | Host | platform2/vm_tools/9s | dev-rust/9s |
| chunnel | Host, Termina | platform2/vm_tools/chunnel | chromeos-base/chunnel, chromeos-base/termina_container_tools |
| cicerone | Host | platform2/vm_tools/cicerone | chromeos-base/vm_host_tools |
| concierge | Host | platform2/vm_tools/concierge | chromeos-base/vm_host_tools |
| container .debs, Termina build scripts | Container | platform/container-guest-tools | N/A |
| crostini_client | Host | platform2/vm_tools/crostini_client | chromeos-base/crostini_client |
| crosvm | Host | platform/crosvm | chromeos-base/crosvm |
| garcon | Termina, Container | platform2/vm_tools/garcon | chromeos-base/vm_guest_tools, chromeos-base/termina_container_tools |
| LXD | Termina | github/lxc/lxd | app-emulation/lxd |
| maitred | Termina | platform2/vm_tools/maitred | chromeos-base/vm_guest_tools |
| seneschal | Host | platform2/vm_tools/seneschal | chromeos-base/vm_host_tools |
| sommelier | Termina, Container | platform2/vm_tools/sommelier | chromeos-base/sommelier, chromeos-base/termina_container_tools |
| system_api | Host | platform2/system_api | chromeos-base/system_api |
| tremplin | Termina | platform/tremplin | chromeos-base/tremplin |
| VM protobufs | Host, Termina, Container | platform2/vm_tools/proto | chromeos-base/vm_protos |
| vm_syslog | Host, Termina | platform2/vm_tools/syslog | chromeos-base/vm_guest_tools, chromeos-base/vm_host_tools |
| vsh | Host, Termina, Container | platform2/vm_tools/vsh | chromeos-base/vm_host_tools, chromeos-base/vm_guest_tools, chromeos-base/termina_container_tools |
Ensure you are able to SSH to the device:
(inside) $ export DEVICE_IP=123.45.67.765 # insert your test device IP here (inside) $ ssh ${DEVICE_IP} echo OK
For the rest of this document, it will be assumed that the BOARD environment variable in cros_sdk is set to the board name of your test device as explained in the Select a board section of the Chromium OS Developer Guide.
Crostini requires a signed-in, non-guest user account to run. You can either use a test Google account, or run /usr/local/autotest/bin/autologin.py -d on a test image to log in with a fake testuser profile.
To begin working on a change to one of the host services (see Where does the code live?), use the cros_workon command:
(inside) $ cros_workon --board=${BOARD} start ${PACKAGE_NAME}
Now that the package(s) are cros_workon start-ed, they will be built from source instead of using binary prebuilts:
(inside) $ emerge-${BOARD} ${PACKAGE_NAME}
Then deploy the package to the device for testing:
(inside) $ cros deploy ${DEVICE_IP} ${PACKAGE_NAME}
Most VM services on the host have upstart conditions start on started vm_concierge and stop on stopped vm_concierge. Since restarting only one daemon could lead to inconsistent state across all services, it's best to shut them all down, then start them again with a clean slate. Stopping the vm_concierge service on the host stops all running VMs, along other VM services on the host. Starting vm_concierge will trigger other VM services to start as well.
(device) # stop vm_concierge && start vm_concierge
The guest packages that run inside the termina VM are built for two special Chrome OS boards: tatl (for x86 devices) and tael (for arm devices). These VM images are distributed as part of the termina-dlc DLC package.
To determine the guest board type, run uname -m on the device.
uname -m | termina board |
|---|---|
x86_64 | (inside) $ export GUEST_BOARD=tatl |
aarch64 | (inside) $ export GUEST_BOARD=tael |
First, cros_workon --board=${GUEST_BOARD} start each guest package you are modifying (see Where does the code live?):
(inside) $ cros_workon --board=${GUEST_BOARD} start ${PACKAGE_NAME}
This is the equivalent of cros deploy for host OS packages, it will deploy modified files for a package. Note that this just copies files and won't e.g. deploy dependencies. Each time you sync you'll usually need to deploy a full image first, then can deploy individual packages.
Build the package like a normal CrOS board:
(inside) $ cros_workon_make --install --test --board=${GUEST_BOARD} \ ${PACKAGE_NAME}
Then deploy it like a normal CrOS package, except using a different tool. --restart-services will shut down all running VMs so your changes are picked up
(inside) $ ../../chromite/contrib/guestos/deploy_to_termina \ --board ${GUEST_BOARD} ${DEVICE_IP} ${PACKAGE_NAME} --restart-services
This is the equivalent of cros flash for a host OS image, it will rebuild and redeploy everything and is needed for some changes (e.g. adding new users), but is slower.
Build the guest image like a normal CrOS board:
(inside) $ ./build_packages --board=${GUEST_BOARD} (inside) $ ./build_image --board=${GUEST_BOARD} test
This image is installed into the host image by the termina-dlc package, and can be built and deployed like the host service changes above:
(inside) $ cros_workon --board=${BOARD} start termina-dlc (inside) $ emerge-${BOARD} termina-dlc # Shut down any running VMs before deploying for changes to get picked up. $ ssh ${DEVICE_IP} -- restart vm_concierge (inside) $ cros deploy ${DEVICE_IP} termina-dlc
After cros deploy completes, newly-launched VMs will use the test DLC with the updated packages.
Packages can end up in the container by two mechanisms:
Native Debian packages (.debs) are preinstalled in the container, and upgraded out-of-band from the rest of Chrome OS by APT.
Packages built from Portage in Chrome OS are copied into /opt/google/cros-containers in Termina by the termina_container_tools ebuild. These are updated with the Termina VM image.
When working on Debian packages, the .debs should be copied to the crostini container and installed with apt:
# A leading "./" or other unambiguous path is needed to install a local .deb. (penguin) $ apt install ./foo.deb
Portage-managed packages should be treated like other [Guest service changes]. However, the termina_container_tools package is not cros_workon, so it must be manually emerged to propagate changes into /opt/google/cros-containers. The following example uses sommelier:
(inside) $ emerge-${GUEST_BOARD} sommelier # build for Termina (inside) $ emerge-${GUEST_BOARD} termina_container_tools # copy into /opt
Once termina_container_tools is manually rebuilt, the termina-dlc flow will work as normal.
It's possible to run binaries built for the termina VM from the chromium OS chroot, which can be useful for debugging or testing. Assuming you already have a chromium OS chroot set up and have built the tatl board, you can run inside the chroot:
../platform2/common-mk/platform2_test.py --board tatl [--run_as_root] [command to run]
For example, to run LXD, you could run inside the chroot:
../platform2/common-mk/platform2_test.py --board tatl --run_as_root env LXD_DIR=/path/to/lxd/data lxd ../platform2/common-mk/platform2_test.py --board tatl --run_as_root env LXD_DIR=/path/to/lxd/data lxd waitready ../platform2/common-mk/platform2_test.py --board tatl --run_as_root env LXD_DIR=/path/to/lxd/data lxc [lxc subcommands go here]
This is not limited to x86_64 boards either, platform2_test.py will automatically set up QEMU to run ARM binaries if you ask it to run binaries from an ARM board. This is likely to be very slow, however.
Note that this is not equivalent to actually running the VM, since only the command you run will be performed. Depending on the exact set up you need to test, this may not be sufficient.
For all the below commands, replace USER_HASH with the CrOS user hash. You can set this with the following command:
export USER_HASH=$(ssh $DEVICE -- dbus-send --system \ --dest=org.chromium.SessionManager --print-reply --type=method_call \ /org/chromium/SessionManager \ org.chromium.SessionManagerInterface.RetrievePrimarySession \ | awk 'NR==3' | cut -d "\"" -f2)
You can log into a running termina from a remote device using vsh via ssh:
ssh $DEVICE -t -- vsh --vm_name=termina --owner_id=$USER_HASH --user=root -- LXD_DIR=/mnt/stateful/lxd LXD_CONF=/mnt/stateful/lxd_conf
To log in as chronos instead of root remove the '--user-root'. The LXD_* lines are to set up the environment so you can talk to LXD.
Copying files is a bit hackier, but you can use the following:
# Local to VM cat $FILE | ssh $DEVICE -- vsh --vm_name=termina --owner_id=$USER_HASH --user=root -- cp /proc/self/fd/0 $DESTINATION # VM to Local ssh $DEVICE -- vsh --vm_name=termina --owner_id=$USER_HASH -- cat $FILE > $DESTINATION
If you want to copy a directory, sorry, you'll have to do it file-by-file or tar it first.
You can log into the container similarly to the above by running
ssh $DEVICE -t -- CROS_USER_ID_HASH=$USER_HASH vmc container termina penguin
Since root-in-termina has access to the container filesystem you can copy files into the container the same way, with $DESTINATION being somewhere inside /mnt/stateful/lxd/storage-pools/default/containers/penguin/rootfs/.
The above are somewhat limited (e.g. can only copy one file at a time). You can also set up sshd inside Crostini and use it for ssh, scp, etc.
You can now connect to Crostini the same as you would the host.
Host service changes can be debugged the same as any other host service. See the main developer guide for details.
In theory you could build gdb into the termina image and use that, or build sshd and gdbserver in and somehow use that. If anyone does manage to debug termina services please update this.
You can install and run gdb inside the container for most things, if you get an error try sudo gdb. Anything that's built inside the CrOS chroot can be debugged using gdb-$BOARD, the same as a host service. This requires some setup but will get you symbols and source code.
One-time setup:
sudo apt install gdbserver binutilsNow you can follow the instructions in the main developer guide to debug chroot-built services inside Crostini the same as you would on the host (except use tatl/tael as your board).
Extra-steps for services running under ld-linux:
gdb cannot automatically load symbols for any of the binaries in /opt/google/cros-containers; for example, Sommelier. This is because these binaries use ld-linux to load alternative versions of dynamic libraries. We can work around this by loading symbols manually.
For each library or binary that's not recognised run the following in a shell on the device (tip: gdb has a shell command):
base=$(cat /proc/$PID/maps | grep $BINARY_PATH | head -n1 | cut -d- -f1) offset=$(readelf -WS $BINARY_PATH | grep \.text) | tr -s ' ' | cut -d' ' -f6) echo Command is "add-symbol-file $BINARY_PATH 0x$base+0x$offset"
Then run the given command in GDB to load symbols for $BINARY_PATH.