Jasmine Tang

Prologue

Hi everyone, how is everyone doing? Gosh, it's been such a long time since my last post with Nix. I hope everybody's doing ok. This May I've moved to Somerville, MA, and then I've also recently moved back to OC, California this July. Things have been really crazy.

This blog is about the time I learned how a caching system works, how much I had improved sccache :) I hope everybody enjoys.

Here are this article's music recommendations:

Cơn Bão Nghiêng Đêm and https://youtu.be/LpXWfvun7DM?si=qj_UdBJTGREwUaKx

Introduction

Caching is an integral part of the incremental workflow: a caching binary needs to process a compiler's command and then figure from the command, which part it needs to cache and how it's gonna cache it. While helping set up the necessary toolchains to build a large scale production app, I've found that caching on a sandbox-based build system is extremely tricky and performance-restrictive. This blog, then, is about my journey of improving the performance of sccache under such a build system.

Here are some notes for readers.

Note:

Note that in mentioning the usage and speedup of sccache here, I mean that the codebase being built cannot utilize the native built-in caching of a said build system, thus having to rely on sccache. There can be a myriad of reasons for such a case. In our case, the reason is that the codebase's build rules are not written in the said build system, and we opted not to build the repository natively due to the maintenance cost. The sandboxing build system ends up invoking the foreign build as a single opaque step, so a single modification to the source code forces us to recompile the whole repository. Thus the need for the external sccache.

Note that SCCACHE_BASEDIRS is an already existing feature in upstream sccache. Also note that partial duplicate work regarding strengthening of SCCACHE_BASEDIRS (though not aimed at sandboxed build systems) is being posted on sccache with https://github.com/mozilla/sccache/pull/2711. To the best of my knowledge, at the time of downstream sccache implementation, this has not yet been posted.

In this article, when we say "clang," we usually mean the gcc/clang compiler usually used in compilation.

Also, in this article, when we say a build system has sandboxing capabilities, we mean on every compilation invocation or build steps, the build system decides to copy each build component to a randomly generated path to compile a build object.

The problem

When we put sccache onto sandboxing build systems, what often happens is sccache will not be able to opt into its direct mode at all, having to fall back to non-direct mode because the sandbox path tangled within each hash key keeps changing.

Note that for context here, when we talk about non-direct mode versus direct mode, we mean direct as: "Can sccache skip the preprocessing step, by directly hashing the related ingredients that will influence a compiler command?"

In non-direct mode, sccache, having to rely on the clang's preprocessor, need to spawn the clang executable, slowing down the system via page-faults, can be detrimental to the caching speed of a build system. Measuring on internal codebase shows that sccache's non-direct mode can be twice as slow (or more) as direct mode.

The fix

For background information, here's how sccache works. The first time sccache's invoked, it spins up a server; this is the sccache server. Later local invocations of sccache, trying to speed up the compilation process, talk to the sccache server to receive the build result instead of calling clang and waiting for the result.

Right now, sccache, when caching its preprocessing output, always caches the full path of its header files. For example, if your header is /usr/lib/yourmom.h, then what is stored in the preprocessor cache is also /usr/lib/yourmom.h. Another example, if your header is ~/Developer/Igalia/projectA/inc/yourmom.h, then it'll also be stored as such as well.

Now, in the case of a sandboxing system, when building an action, supposing the same project path, the header path being cached will be something of the effect of /build_system/ssahdgfkajsdgfkasfgsak/inc/yourmom.h.

This means that for direct mode, since the same dependency being built will be spawned in different paths each time, and that each sandbox will be constructed and torn down after every build action is completed, this will be devastating for sccache's direct mode. It cannot utilize the cached preprocessor output anymore, since the cache key of said cached preprocessor output will be nonexistent (torn down already).

The following fix, relying on the assumption that there can only be a single element in a SCCACHE_BASEDIRS list supplied from the sccache client, is an attempt against such shortcomings in sccache direct mode.

To be short: strip the sandbox prefix sent in via the sccache client from all paths before hashing, so cache keys are identical across sandboxes, and then re-add the prefix when checking whether the include files exist in the sandbox directory.

On the client side

Often, in an extensible build system, we can query the sandboxing path that a build system's gonna use to build a build object.

Suppose the name of the sandboxing path that we query for a build action is SANDBOX_PATH, we would need to construct a sccache_launcher so that we can inject the environment variable SCCACHE_BASEDIRS="$SANDBOX_PATH" to the invocation of the sccache client.

With respect to the source code of sccache, there won't be any modification needed for the client.

On the server side

On the sccache server side, we would need to make some code modifications to the source code :)

Now, an interesting fact about the compilation request that the sccache client sends to its server is that it carries with it the environment variables of the client (Hence the sccache_launcher in the client modification side)

First, in generate_hash_key() of src/compiler/c.rs, we would need to intercept the compilation request to fish out the SCCACHE_BASEDIRS environment and merge it with the server's empty SCCACHE_BASEDIRS. This is also a good time to set some assert!() to make sure there is only one singular path in all of SCCACHE_BASEDIRS.

Now, in the same file src/compiler/c.rs, we would need to also influence how the preprocessor output is hashed in impl HashKeyParams's compute(). More specifically, we would need to strip out the path from its arguments; these are often -I arguments that carry the sandbox path that's subsumed by the sccache client's SCCACHE_BASEDIRS. Now, repeat the same step for how the preprocessor cache's hashed in src/compiler/preprocessor_cache's preprocessor_cache_entry_hash_key.

After we've stripped the base dirs out of all the arguments, we would then also need to strip the base dirs out of the include files in src/compiler/preprocessor_cache.rs's add_result(). Since include files are one of the cache keys to skip the preprocessing step, if we don't do this, caching will never hit because each computed hash key will be different, coming from a new sandbox path each time. Needless to say, this principle also applies to the previous paragraph.

Finally, since we always strip the base dirs out of everything, we would need to add them back when sccache decides to check if include files are there or not in src/compiler/preprocessor_cache.rs's result_matches().

Performance considerations

Voilà! You're done, now when you've recompiled the code, you'll be able to enjoy the sweet sweet improvement. This section then discusses the performance improvement of such code modification.

For a random repository Ada (think of the big size of LLVM and such), here are such measurements on configure and make:

• Without sccache: 170 seconds.
• With sccache non-direct mode: 68 seconds.
• With sccache direct mode: 35 seconds.

Now, in profiling the build system, I observe that configure takes about 15 seconds each time, giving the following true build time:

• Without sccache: 155 seconds.
• With sccache non-direct mode: 53 seconds.
• With sccache direct mode: 20 seconds.

This means that with our patch, we managed to reduce the build time of a big repository by 7.75 times if we don't use sccache to start with, and 2.65 times if we had already used sccache to begin with.

Yayyyyyyyy! Mission accomplished :)

Pitfalls and Going further

What's slow in sccache (non-direct mode)

Now, it's probably good to understand what's slow in the non-direct mode. In profiling the build process for a big repository, I've observed that a lot of time is spent loading the clang executable (taking page faults in the process due to the usual massive size of the clang executable) and initializing the preprocessor, using the result as a key to the build object itself. sccache direct mode bypasses all this, removing a lot of time spent in clang and its preprocessor. Here are a few other things you can do other than enabling direct mode.

Alleviating page faults

Now, in my case, it was easier to clone sccache and modify it downstream to make direct mode work in a sandboxing build system than it was to influence the release process of the clang executable used downstream. If you are able to, you can also build only the needed components of clang so that the executable size stays smaller, alleviating the page fault hits if you have to rely on the preprocessor.

jobserver

If you truly want to take this a step further, you can look into how sccache uses its jobserver. In its configure() stage in .rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std /src/sys/process/unix/common.rs, it performs a pre_exec, which pushes a closure onto the Command struct. When it does this, posix_spawn is not used and is replaced by fork + exec(), which, for some reason, makes sccache a tiny little bit slower.

You can modify the sccache source code in src/jobserver.rs to create the jobserver as a named FIFO at /tmp/sccache-fifo-server to bypass this requirement and let std use posix_spawn. The function to modify is new_num().

musl malloc

Suppose somebody is really passionate about reducing build time, one can take a look at how sccache is packaged.

In their releases, the sccache repo releases the binary built with musl malloc, which installs a single global lock and can slow down your sccache processes. Switching to glibc will help with this.

Note that if you use mimalloc (which is another non-single global lock malloc library), you'll need to either disable transparent huge pages or modify sccache to not use fork + exec anymore (see the jobserver section).

Performance consideration (again)

Now, after you've done all this, you'll notice the build time of your project reduces a bit more. In my measurement of the build process, the build time of repository Ada drops from 35 seconds to 30 seconds.

This means that the improvement over the true build time is now:

• x10.33 if you weren't using sccache before.
• x3.53 if you were using the prebuilt sccache in non-direct mode.

What to use as the sccache_launcher

I want to talk about the choice of languages used to write the sccache_launcher. The first time the sccache launcher was written, I wrote the launcher in a bash script. After seeing how slow bash was iterating over its passed-in arguments compared to Python, I then proposed to the team that we switched to Python.

What I didn't realize was that although Python iterates on the argument list much faster than bash, it takes a lot more time to start up to actually run the script. With the enormous number of invocations to the sccache_launcher, it turns out the Python launcher was so slow, it was actually as slow as when we don't have sccache involved at all. The dangerous thing is that if you think the problem was caused by sccache itself, you won't be able to find any clues at all; these Python sccache_launcher are tiny short-living invocations that's repeated hundred thousand times, stacking up its inefficiency little by little, impossible to perf or trace. I basically have to git bisect all sccache-related patches to finally get the culprit. It turns out the perpetrator was me the whole time hahahah.

My main take-away from this is two folds:

• Small benchmark gains of sccache don't equate to real performance gains on your build system. Don't let your engineers merge in the caching code without having them retest it on the whole build process one last time.
• A small setup to monitor your build system's health and speed would probably do us a lot of goods here. Even a dashboard graph showing the percentage decrease or increase in build time would help us identify the problem so quickly.

Epilogue

Anddd that's it. I hope everyone enjoyed the article. I've really enjoyed the opportunities to work on sccache and learn more about profiling via perf and performance engineering at Igalia :)

See everybody next time.