Jasmine Tang

History and motivation

ccache (and it's sccache) is an amazing piece of technology. By caching compiler invocations, it's saved who knows how many hours of computation.

In ccache version 3.0, Joel Roshdal came out with the release of direct mode, which offers a significant boost to the caching speed by bypassing the preprocessor step itself, relying only on the compiler command itself, the header files being used and the source file being compiled.

Anecdotal experience shows that by going from no caching at all to nondirect mode (re-invoking the preprocessor of a compiler), we can observe a 3x speedup; if we're going further and opt into direct mode, we can see a x10 speed up from base performance.

However, over the years, there's been multiple discussions on the stale cache issue of ccache. For example, see https://ccache.samba.narkive.com/nqHl8wLK/direct-mode-design-bug, filed by the ccache's author Joel Rosdahl.

The core of the issue is, "ccache records the header files that were used by the compiler, but it doesn't record header files that were not used but could have been used if they existed. So, when ccache checks if a result could be taken from the cache, it can't check if the existence of a new header file should invalidate the result."

The 3 proposals in the original post doesn't seem to be feasible due to the needed engineering and maintenance effort. In a private discussion with the maintainer, there's an agreement that reusing the preprocessor output during the cold caching stage seems like the best way going forward. More specifically, by appending the -v argument to the compiler's preprocessor output.

Seeing that sccache is a different implementation of ccache in Rust that still borrows direct mode's idea from ccache, this article then discusses the steps to get sccache and ccache working under a new mode: safe direct mode, that addresses the shortcomings of direct albeit limiting a bit of performance of the original direct mode.

General steps

Adding -v to preprocessor

So, how can we fix this? First, we'd need more info out of a compiler's preprocessor's output. Besides the usual header files being used being recorded, we'd also want to know what include folders the compiler decides to search in. All of this can be done through appending the verbose optino to the compiler's preprocessor

Knowing what include folders the compiler cannot search in is also very helpful. Again, the principle cause is that "it (ccache) can't check if the existence of a new header file should invalidate the result". A new header can come in from two ways:

1. It comes from a include dir that exists, but is lower on the search list priority, so the compiler doesn't search in it.
2. It comes from a include dir that doesn't exist, so the compiler can't search in yet.

Setting up the include-direcotry parser for the preprocessor output

Suppose we have a folder with the source tree as follows:

When we invoke the preprocessor (as ccache and sccache will invoke it), the compiler's preprocessor output will show some output such as the aforementioned picture.

There are a few elements to note here for the parser:

• The parser would need to parse the include directory not just from the line #include "..." search starts here: and #include <...> search starts here: (representing case 1 of the previous section), it would also need to take care of the second case, meaning the lines starting with ignoring nonexistent directory

TODO: Jasmine ask AI and reconfirm if there's a standard on preprocessor output

Figuring out what directory can contain which header

Now, there are some distinctive differences between ccache and sccache that we should be careful about.

For example, sccache's direct mode is a bit less mature than ccache, and their path handling is not sophisticated enough (Todo rephrase this better i just mean sth about absolute path).

We should also be mindful of ccache's current work around of the issue. Right now, it attempts to fix the issue by reading the argument list of the compiler invocation and track if a header file can be in one of the folders. Of course, relying on heuristics only and not reusing the fullness of the preprocessor will cost a few accuracy here and there.

sccache

ccache