Original link: https://xuanwo.io/reports/2022-50/
Very happy recently!
At the end of Weekly 2022-45: Using sccache to speed up Rust compilation , I mentioned that I hope to replace the storage backend of sscache with opendal :
I submitted a proposal to the upstream: Use opendal to handle the cache storage operations to see if I can change the storage backend access of sccache to use opendal, so that it will be more convenient to connect storage and debug~
Now it’s real! Today’s weekly report will talk about the story of Sccache and OpenDAL.
Dissatisfied with the status quo
In order to quickly launch the compile cache function in Databend’s CI, I forked sscache and made some modifications, but in the process of modification, I found that it is far more efficient to repair the existing codebase than to go directly to opendal. So I spent some time modifying sscache’s s3 storage backend implementation to opendal. Thanks to opendal’s good log and other infrastructure, I quickly located some problems on the AWS configuration side and went online smoothly. The process and final output have been discussed in the 2022-45 weekly report. In order to facilitate online CI use, I modified Cargo.toml and pushed cccache .
After the compilation cache went online, it ran very stable, so I decided to continue using it like this. But this will face a problem of long-term maintenance in the future: I don’t know anything about compilation, and it is definitely impossible to maintain a project like sscache for a long time. Not to mention implementing more functions, just dealing with possible bugs is a headache. The best way is that I can feed back the current modification to the upstream, so as to ensure that my service can run stably for a long time.
put forward a draft
So I proposed a draft: proposal: Use opendal to handle the cache storage operations . In the draft, I briefly introduced what opendal is, and introduced the benefits of using OpenDAL:
- Can fix many existing bugs (the implementation of s3/gcs/azblob written by hand in the past was relatively simple, and many details were not considered)
- Reduce the burden on maintainers (hand over the specific service implementation logic to opendal, and sscache no longer needs to worry about the implementation details of storage)
- It is easier to implement service support (for example, the community wants native OSS support)
Feedback from the maintainer was neutral, with some questions about performance and what current OpenDAL users have. In terms of performance, I used sscache and cccache to do benchmarks respectively (I thought it should be similar, but cccache is faster than sscache, I don’t know why). I found that maintainers are more concerned about which users of OpenDAL are currently using them. They don’t want their projects to depend on a project that will be yellow in a few months. From a maintainer’s perspective, the long-term maintenance commitment of a dependency is often more important than a small short-term advantage. The exact words of @sylvestre are:
Sure, I just don’t want to add a critical dependency which could be unmaintained in a few months
Take the initiative
After receiving my reply, the maintainer did not respond. I boldly decided to take the initiative: I submitted a PR to use OpenDAL to replace the implementation of s3. The intention was to show my muscles and prove that using OpenDAL is really easy and can Reduce maintenance burden. It’s a pity that this Demo PR did not pass CI, because OpenDAL requires at least Rust 1.60 to build. I was a little frustrated, but the maintainers were really nice, and after asking what the OpenDAL MSRV is, they raised the sscache MSRV to 1.60. Later, OpenDAL’s Demo successfully passed all CIs, and then we waited for the maintainer’s feedback.
After waiting for many days, another maintainer of the project @drahnr came forward to give a formal response, pointing out the risks of using OpenDAL, raising some open questions, and summarizing some benefits of using OpenDAL, The overall attitude is biased toward support.
Some of the risk points and problems raised here are common problems faced by open source projects. I will list them one by one and talk about my responses:
Opendal has a lot of backends that sscache doesn’t need, such as rocksdb.
This is easier to solve. After all, opendal itself uses feature flags to ensure that these backends with additional dependencies will not participate in compilation. I explained this behavior in my response and ensured that sscache is not affected by these unused backends.
How to ensure the follow-up long-term maintenance of opendal?
To be honest, I don’t know how to answer this kind of question better. After all, long-term maintenance is done, and what is said by mouth is not reliable. What’s more, I have experienced the tragedy of the layoff of the beyondstorage team and the suspension of the project, so I am very afraid of such problems.
Finally, my answer is to frankly say the current situation. OpenDAL is the core dependency of databend, which is maintained by the datafuselabs team for a long time, and I mentioned that we are planning to grow our community to attract more maintainers.
How to ensure the stability of the API? Does it follow the principles of semver?
This OpenDAL has been doing well. The release version fully follows the principle of semver, and also maintains the CHANGELOG . I feel that this is a big plus.
How are MSRVs handled?
After the Demo PR overturned, I quickly set up MSRV for OpenDAL and added it to CI, which is used here.
I showed off OpenDAL’s MSRV Check CI , and gave a promise not to modify MSRV before 1.0 .
After I responded to all the above questions, the maintainers clicked their Approve~
current progress
After migrating to s3, I implemented the support of azure and gcs one after another:
It is worth mentioning that these two PRs have deleted a lot of code, and now the implementation of adding a cache service to sscache is as simple as just a few lines:
pub struct AzureBlobCache ;
impl AzureBlobCache {
pub fn build (connection_string: & str , container: & str , key_prefix: & str ) -> Result < Operator > {
let mut builder = azblob::Builder::from_connection_string(connection_string) ? ;
builder.container(container);
builder.root(key_prefix);
Ok (builder.build() ? .into())
}
}
Next, we plan to add OSS native support to sscache~
When sccache is released next time, I can completely kill cccache in databend CI and replace it with upstream sscache. It is said that sscache will also be used in Firefox CI. I am looking forward to it, and I feel that it is really changing the world.
Afterthoughts
- Show Me The Code: The actual code is more impressive than the abstract Proposal, so when mentioning this kind of large-scale refactoring, attaching a simple PoC to illustrate your own ideas is often easier to get support from maintainers
- The value of long-term maintenance: project maintainers often care more about the long-term maintenance commitment of the project than the short-term advantages and Fancy features of the project
- Code style and documentation: These are long-term behind-the-scenes work, but they will become very important considerations when making project decisions. One thing Sccache values in OpenDAL is that the rich documentation and code are much better than the existing ones
After Datbaend / Greptime / Sccache, if OpenDAL can find two more big users in the community, and find a few Committers, I feel that I can start donating to Apache. I am looking forward to it.
rush!
This article is transferred from: https://xuanwo.io/reports/2022-50/
This site is only for collection, and the copyright belongs to the original author.