Add ccache support to Dockerfiles and build workflow by lukovdm · Pull Request #892 · stormchecker/storm

lukovdm · 2026-03-19T11:43:41Z

The PR CI actions where slowing me down quite a bit thus I investigated a bit into how it could be sped up. I added caching of the homebrew downloads and added ccache in all builds with a github cached ccache.

…ld performance

lukovdm · 2026-03-19T16:03:16Z

It seems like using ccache together with precompiled headers (PCH) does not really work. You get a really low ccache hit rate like this:

Cacheable calls:                     57 / 649 ( 8.78%)
  Hits:                              57 /  57 (100.0%)
    Direct:                          57 /  57 (100.0%)
  Misses:                             0 /  57 ( 0.00%)
Uncacheable calls:                  592 / 649 (91.22%)
  Could not use precompiled header: 592 / 592 (100.0%)

I have done a quick benchmark (on a 24 core machine) of turning PCH on and off with and without a warm cache. These are the results:

PCH	cache state	build (s)	cacheable calls	hits	misses	uncacheable calls
on	cold	178.33	57/649(8.78%)	0/57(0.00%)	57/57(100.0%)	592/649(91.22%)
on	warm	172.48	57/649(8.78%)	57/57(100.0%)	0/57(0.00%)	592/649(91.22%)
off	cold	200.38	648/648(100.0%)	0/648(0.00%)	648/648(100.0%)	0
off	warm	56.97	648/648(100.0%)	648/648(100.0%)	0/648(0.00%)	0

So with a cold cache disabling PCH costs time. But with a warm cache all compile calls actually hit the cache, and it is thus much faster. Most time was taken up by the dependencies.

For the GitHub actions CI we would always have a warmish cache, and thus it seems like it would be worth it to me to disable PCH for them.

For people building storm themselves I don't know, it depends on how much we value cold cache builds versus warm cache builds.

…kflow; add benchmark script for compile time analysis

volkm · 2026-03-20T08:20:15Z

Nice idea @lukovdm! I think it is good to try to decrease the building times in the CI.
I like the idea of using a cache to have incremental builds.

Some thoughts:

I would probably opt to use the caching for PRs and branches other than master. That way, all feature branches have faster CI by incrementally building new commits. Seeing that most PRs nowadays have multiple iterations, this would decrease the computation times. On the master branch, I would still build from scratch to catch potential issues not detected by the incremental build. In general, I would also run more tests on the master than on the feature branch.
Interesting to see that PCH have such a negative influence on the cache. Could this be somehow mitigated? I briefly googled and it seems that it should not be a general issues (at least people implemented fixes throughout the years).
Do you see any cache issues with dependencies? On my local machine, it sometimes feels to me that a large rebuild is triggered even though not much has changed. I suspect a bit that some of the 3rdpary dependencies could have an influence.
In CMake, I suggest to add a new option STORM_COMPILE_WITH_PCH which allows to enable/disable the PCH. This is similar to STORM_COMPILE_WITH_CCACHE. Edit: According to precompiled headers #451 there is cmake_disable_precompiled_headers
Why are all the includes necessary now? It is interesting that it compiled before.

lukovdm · 2026-03-20T08:32:55Z

Thanks for the questions.

The includes are necessary I think because https://github.com/lukovdm/storm/blob/fc5220f3e1920dab08030f4d47c00111e1883250/CMakeLists.txt#L315 includes several headers for many files.
About the PCH, looking at these docs, using PCH and pragma once might be the issue: https://ccache.dev/manual/latest.html#_precompiled_headers. But I did not want to change pragma once in all files. I tested a several things to get PCH and ccache to properly work but could not do it.
The flag and branch things I can implement.
I have not noticed anything with the dependencies, but they are the slowest part of a warm cache clean build folder build now.

Also, some GitHub workflows are not properly caching yet, but the once that are, are running in 4-10 minutes.

volkm · 2026-03-20T09:23:38Z

Thanks for the detailed answers.

One thing I noticed looking at the latest CI run, for example here is that carl and sylvan seem to be rebuild each time. I think that might be due to the way how we configure the fetchcontent.

sjunges · 2026-03-20T10:19:40Z

Thanks Luko!

Do you have any profiling of the compilation process with a warm start? We do have some ways of profiling storm compilation processes and it would be nice to know where the time is lost.

I second Matthias: For the main branch, PCH seems preferable, in particular as I care about the cold start compile time a lot. (Although this is shifting with more people using binaries).
Should we use cmake_disable_precompiled_headers in the STORM_DEVELOPER mode?

If your goal is to speedup the CI, I think it also really gets time to exclude some tests --, for runs where the CI is failing during compilation, that probably doesn't matter that much, but we sometimes have to wait for CI as we cannot run too many things in parallel...

volkm · 2026-03-20T10:34:44Z

I opened a discussion for a CI revision #894

volkm · 2026-03-20T10:37:37Z

Related to #763

lukovdm · 2026-03-20T12:21:21Z

I got some profiles.

Ignoring the dependencies, with a cold cache, with PCH takes 3m40s, without PCH takes 4m25s. This difference is caused by most files taking a few extra seconds, not one that takes way longer.
A warm Ccache and PCH results in all dependencies except spot to be compiled basically instantly. Spot is not affected.
Warm ccache without pch is dominated by spot:

volkm · 2026-03-20T12:46:37Z

Thanks for the investigation. Looks like spot will be recompiled any time. I think we should figure out why this is happening and whether we can prevent this.

lukovdm · 2026-03-20T15:52:29Z

I got spot to use ccache and this helped quite a bit again. These benchmarks also have ass tests enabled:

pch	cache_state	build_seconds	cacheable_calls	hits	misses	uncacheable_calls
off	cold	331.26	1558/2004(77.74%)	81/1558(5.20%)	1477/1558(94.80%)	446/2004(22.26%)
off	warm	22.46	1528/1961(77.92%)	1528/1528(100.0%)	0/1528(0.00%)	433/1961(22.08%)
on	cold	286.60	526/2007(26.21%)	24/526(4.56%)	502/526(95.44%)	1481/2007(73.79%)
on	warm	239.90	496/1964(25.25%)	496/496(100.0%)	0/496(0.00%)	1468/1964(74.75%)

And warm cache with PCH off:

…ectly

sjunges · 2026-03-20T19:57:14Z

Ok, what I gather from this are a few questios:

The tests are still really slow. I know I never optimized for them, but why do they not profit from the cache.
I wonder whether there is more that can be compiled while spot is being built.
Spot configure seems to be one of the main bottlenecks.

sjunges · 2026-03-20T20:01:20Z

Regarding spot: We could

use system-versions of spot again
or we could cache the configure step? (autotools supports cache for the configure step).
There also seem to be a few a configure options (such as disabling python bindings) which probably are useful.

volkm · 2026-03-23T09:54:16Z

The time for the tests could be mostly the linking time? But ideally they do not need to be rebuild if nothing changed.
The storm CMake target requires the resources target which contains spot. Other resources such as Cudd, sylvan can be build in parallel with Spot, but Spot takes significantly longer than those.
The Python bindings should already be disabled.

Things to try:

System version of Spot: we could install spot via homebrew and try to install the Debian-based packages for Debian and Ubuntu (https://spot.lre.epita.fr/install.html). We currently only use preinstalled Spot in the Docker storm-dependencies.
Caching the configuration steps could be a good option. But in principle, the reconfiguration should only be run if things changed anyway. So it would be good to figure out why it is reconfigured in the first place.

…om different sources. Currently we have the old every PR commit trigger and a PR label trigger. They can have different configs.

sjunges · 2026-04-16T13:34:58Z


 # Build Storm
 #############
+RUN if ! command -v ccache >/dev/null 2>&1; then \


Can you write some comments here to clarify what this does (and why) for other users.

Also can we make the usage of ccache configurable?

sjunges · 2026-04-16T13:36:26Z

Code/build changes LGTM; @volkm can you go over the github action changes?

…lows

…ModelChecker

volkm

Thanks for tackling this undertaking of revising our CI. I like the new structure and it should decrease the time for CI and make maintenance a bit easier. Great work!

I have a number of comments, I hope most are just small ones.

volkm · 2026-04-16T14:18:27Z

    set(STORM_DEBUG_SPOT ON)
    set(STORM_DEBUG_SYLVAN ON)
    set(STORM_WARNING_AS_ERROR ON)
+	# Turn off PCH for faster ccache usage on warm caches.


Suggested change

# Turn off PCH for faster ccache usage on warm caches.

# Turn off PCH for faster ccache usage on warm caches. PCH compiled files cannot be properly cached currently.

volkm · 2026-04-16T14:19:18Z

-		set_property(GLOBAL PROPERTY RULE_LAUNCH_LINK ccache)
+
+		if(STORM_COMPILE_WITH_PCH)
+			message(STATUS "STORM - PCH is enabled which will lead to significantly lower ccache hit rates. Consider disabling PCH if you plan to rebuild storm multiple times.")


Suggested change

message(STATUS "STORM - PCH is enabled which will lead to significantly lower ccache hit rates. Consider disabling PCH if you plan to rebuild storm multiple times.")

message(STATUS "STORM - PCH is enabled which will lead to significantly lower ccache hit rates. Consider disabling PCH if you plan to rebuild Storm multiple times.")

volkm · 2026-04-16T14:19:48Z

 	mark_as_advanced(CCACHE_FOUND)
 	if(CCACHE_FOUND)
 		message(STATUS "Storm - Using ccache")
+		set(ENV{CCACHE_SLOPPINESS} "pch_defines,time_macros,include_file_mtime,include_file_ctime")


Can you add an explanations for this selection here?

volkm · 2026-04-16T14:21:46Z

 		"<optional>" "<ostream>" "<istream>" "<list>" "<set>" "<fstream>" "<string>" "<boost/optional.hpp>" "<boost/variant.hpp>"
 		"<boost/container/flat_set.hpp>" "<boost/dynamic_bitset.hpp>" "<boost/range/irange.hpp>")

+function(storm_target_precompile_headers target)


Maybe move to resources/cmake/macros or to src/CMakeLists.txt?

volkm · 2026-04-16T14:25:37Z

 # Build Storm
 # (This can be adapted to only build 'storm' or 'binaries' depending on custom needs)
-RUN make -j $no_threads
+RUN ccache --max-size=3G && ccache --zero-stats && make -j $no_threads && ccache --show-stats --verbose


Can we make the magic number 3G configurable?
Is showing the ccache stats necessary? Users who simply build Storm probably do not care about it.

Why do we actually need a call to ccache? Should STORM_COMPILE_WITH_CCACHE not automatically take care of it?

volkm · 2026-04-21T14:12:57Z

+        type: string
+        default: "Debug"
+      Developer:
+        description: "Enable developer mode (ON/OFF)"


In the interest of nicer handling and especially avoiding invalid input, there could be two options for improvement:

Use custom options "ON"/"OFF" is inputs using input type choice

Use boolean as input and convert to ON/OFF via something like ${{ input && 'ON' || 'OFF' }}

volkm · 2026-04-21T14:24:01Z

+          restore-keys: |
+            buildtest-brew-${{ inputs.distro }}-
+      - name: Ensure ccache directory exists (macOS)
+        run: mkdir -p /Users/runner/.ccache


Suggested change

run: mkdir -p /Users/runner/.ccache

if: inputs.restore_cache

run: mkdir -p /Users/runner/.ccache

volkm · 2026-04-21T14:25:01Z

+          brew update
+          brew install automake boost ccache cln ginac glpk hwloc libarchive xerces-c z3
+      - name: Prepare ccache
+        run: |


Also guarded by save_cache or restore_cache? Also in more places later on.

volkm · 2026-04-21T14:25:52Z

+            --build-arg cmake_args="-DSTORM_WARNING_AS_ERROR=ON -DSTORM_COMPILE_WITH_PCH=OFF"
+            # Omitting arguments developer, disable_*, cln_exact, cln_ratfunc, all_sanitizers
+      - name: Export ccache from Docker image
+        run: |


Should be guarded by save_cache or restore_cache? Also in more places.

volkm · 2026-04-21T14:35:46Z

+  - Summary table in stdout
+  - CSV at <build-root>/results.csv
+  - Per-run ccache stats in <build-root>/results/
+    - Optional traces at <build-root>/results/trace-<pch>-<cache>.json


Suggested change

- Optional traces at <build-root>/results/trace-<pch>-<cache>.json

- Optional traces at <build-root>/results/trace-<pch>-<cache>.json

lukovdm added 4 commits March 19, 2026 12:42

Add ccache support to Dockerfiles and build workflow for improved bui…

a9d9699

…ld performance

cache test

db2357b

Remove PCH when using ccache, fix missing includes.

b565e9a

Update cache key in build workflow and fix includes

d89c26c

lukovdm added 3 commits March 19, 2026 17:07

Enhance ccache usage with verbose output in Dockerfiles and build wor…

d578afc

…kflow; add benchmark script for compile time analysis

Update docker files

c905fc4

warm cache

fc5220f

add PCH flag, disable PCH when developer is turned on

f525bcd

lukovdm added 4 commits March 20, 2026 13:31

Fix Mac CI error with STORM_DEVELOPER

2b66662

Set correct key for Minimal dependices indepth tests

2491036

Disable PCH for linuxtests ci

ddbecae

fix syntax in buildtest.yml

22a84a7

lukovdm added 2 commits March 20, 2026 14:44

Update cache key for build tests to include cmakeArgs

d3eee94

Refine cache keys for build tests to improve cache efficiency

af35ec3

Fix Spot build flags for debug mode and ensure CCACHE is applied corr…

897f8db

…ectly

lukovdm added 2 commits March 26, 2026 14:48

Set workflow cache keys so they are properly saved.

eb9b0a7

Seperate out the build tests in seperate files which can be called fr…

32e0eec

…om different sources. Currently we have the old every PR commit trigger and a PR label trigger. They can have different configs.

lukovdm added 5 commits April 2, 2026 15:22

Merge branch 'master' into cache-actions

65f99d9

Merge build workflows

3db1980

new ci policy

8357a0b

Compact matrices and fix bug in docker base image

07b2b2b

Don't run normal PR CI if ready to review CI PR is used.

8b964cc

lukovdm added the ready for review label Apr 13, 2026

lukovdm added 3 commits April 13, 2026 11:17

Refactor cache keys in test workflows for improved cache management

490f9e4

Merge branch 'master' into cache-actions

c92cbe3

Fix syntax for conditional checks in CI workflow

4c9b11d

sjunges reviewed Apr 16, 2026

View reviewed changes

volkm mentioned this pull request Apr 17, 2026

CI: trigger stormpy tests after changes in Storm #911

Open

lukovdm added 3 commits April 20, 2026 12:19

Fix actions

9d50cc0

Remove unnecessary line breaks in Docker build arguments for CI workf…

9e14620

…lows

Suppress false positive warning from GCC 15 for SparseDtmcElimination…

10013bb

…ModelChecker

volkm reviewed Apr 21, 2026

View reviewed changes

	# Turn off PCH for faster ccache usage on warm caches.
	# Turn off PCH for faster ccache usage on warm caches. PCH compiled files cannot be properly cached currently.

	message(STATUS "STORM - PCH is enabled which will lead to significantly lower ccache hit rates. Consider disabling PCH if you plan to rebuild storm multiple times.")
	message(STATUS "STORM - PCH is enabled which will lead to significantly lower ccache hit rates. Consider disabling PCH if you plan to rebuild Storm multiple times.")

	run: mkdir -p /Users/runner/.ccache
	if: inputs.restore_cache
	run: mkdir -p /Users/runner/.ccache

	- Optional traces at <build-root>/results/trace-<pch>-<cache>.json
	- Optional traces at <build-root>/results/trace-<pch>-<cache>.json

Conversation

lukovdm commented Mar 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

lukovdm commented Mar 19, 2026

Uh oh!

volkm commented Mar 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

lukovdm commented Mar 20, 2026

Uh oh!

volkm commented Mar 20, 2026

Uh oh!

sjunges commented Mar 20, 2026

Uh oh!

volkm commented Mar 20, 2026

Uh oh!

volkm commented Mar 20, 2026

Uh oh!

lukovdm commented Mar 20, 2026

Uh oh!

volkm commented Mar 20, 2026

Uh oh!

lukovdm commented Mar 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

sjunges commented Mar 20, 2026

Uh oh!

sjunges commented Mar 20, 2026

Uh oh!

volkm commented Mar 23, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

sjunges commented Apr 16, 2026

Uh oh!

volkm left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

lukovdm commented Mar 19, 2026 •

edited

Loading

volkm commented Mar 20, 2026 •

edited

Loading

lukovdm commented Mar 20, 2026 •

edited

Loading