2022-45: Speed ​​up Rust compilation with sccache

/ Xuanwo | 漩涡 | Base Tianjin

Original link: https://xuanwo.io/reports/2022-45/

sccache is a compilation and caching tool developed by mozilla . The design idea is to serve as a compilation wrapper, save the compilation product to the storage backend, and avoid repeated compilation as much as possible, thereby speeding up the overall compilation speed. Compilation backends supported by sccache include gcc, clang, MSVC, rustc, NVCC, etc. Taking Rust compilation as an example, it can be used like this:

 export RUSTC_WRAPPER = /path/to/sccache cargo build

Recently, the community asked whether there is a way to speed up the compilation speed of databend. I thought of using sccache to speed up. The advantages are as follows:

  • Databend has many dependencies, and most of them rarely change, and they can be well cached
  • The dev build of Databend runs on the Self-hosted Runner of AWS, and the S3 bucket of the intranet has already been configured.

Start stepping on the pit

The conditions in all aspects are perfect, it seems that you only need to write a Github Action. It turns out I was too optimistic:

There are two main pits:

  • In order to facilitate the maintenance of the test environment, Databend CI uniformly uses the build-tools image to run the test, and it requires a lot of additional configuration when paired with sccache
  • For security reasons, the cloud platform has enabled authentication based on node Role, without configuring AK/SK, and sccache does not support IMDSv2

The first problem was solved by constantly harassing @everpcpc , the latter problem is relatively more troublesome. We first submitted an Issue: Support IMDSv2 for AWS IAM Role authentication to sccache, and then started trying to solve the problem ourselves.

debug sccache

The process of debugging sccache is very painful: sccache is a CS architecture. Every time a user calls sccache, a server will be started in the background, and the communication between the client and the server will determine whether to use the cache. All interactions with the storage backend are performed by the server, and the log cannot be seen directly during the compilation process, but the log can only be redirected to a file, and then the output of the file can be viewed. In addition, sccache has a long history, and many places are not conducive to debugging, such as:

 let res = self  .client  .execute(request)  . await  .with_context( move || format ! ( "failed GET: {}" , url)) ? ;  if res.status().is_success() {  let body = res.bytes(). await .context( "failed to read HTTP body" ) ? ;  info ! ( "Read {} bytes from {}" , body.len(), url2);   Ok (body.into_iter().collect()) } else {  Err (BadHttpStatusError(res.status()).into()) }

When the request fails, only the status code will be displayed, without actual error content, and many detours have been taken when debugging IMDSv2. In order to solve the problem as soon as possible, I directly forked sccache and modified the underlying storage to use opendal:

 - let credentials = self - .provider - .credentials() - .await - .context("failed to get AWS credentials")?; - - let bucket = self.bucket.clone(); - let _ = bucket - .put(&key, data, &credentials) - .await - .context("failed to put cache entry in s3")?; + self.op.object(&key).write(data).await?;

It is much more comfortable now, and the problem was quickly located and fixed. Now Databend has successfully used sccache. After installing sccache, you only need to configure it as follows:

 - name : Setup Build Tool  uses : ./.github/actions/setup_build_tool  with :  image : $  bypass_env_vars : RUSTFLAGS,RUST_LOG,RUSTC_WRAPPER,SCCACHE_BUCKET,SCCACHE_S3_KEY_PREFIX,SCCACHE_S3_USE_SSL,AWS_DEFAULT_REGION,AWS_REGION,AWS_ROLE_ARN,AWS_STS_REGIONAL_ENDPOINTS,AWS_WEB_IDENTITY_TOKEN_FILE  - name : Build Debug  shell : bash  run : cargo -Z sparse-registry build --target $  env :  RUSTC_WRAPPER : /opt/rust/cargo/bin/sccache  SCCACHE_BUCKET : databend-ci  SCCACHE_S3_KEY_PREFIX : cache/  SCCACHE_S3_USE_SSL : true

Summarize

Some relatively simple data are recorded here: when compiling in full for the first time, it takes some extra time to upload the compiled products, and subsequent compilations can reuse these products without compiling.

  • When there is no cache: 6m 20s
  • Compile for the first time
    • Finished dev [unoptimized + debuginfo] target(s) in 9m 04s
  • second compilation
    • Finished dev [unoptimized + debuginfo] target(s) in 5m 35s
  • After removing the debug log
    • Finished dev [unoptimized + debuginfo] target(s) in 4m 38s

The first compilation time here also needs to consider the impact of sccache debug log on performance, in fact, it should not increase that much. However, there are still many places in Databend that cannot be cached. You can see if there are any places that can be optimized later. I submitted a proposal to the upstream: Use opendal to handle the cache storage operations to see if I can change the storage backend access of sccache to use opendal, so that it will be more convenient to connect to storage and debug~

This article is transferred from: https://xuanwo.io/reports/2022-45/
This site is for inclusion only, and the copyright belongs to the original author.