- every entry stood in for a library tsan could not see into; with libzmq,
libsodium and libstdc++ now tsan-instrumented in the tsan CI job, the
happens-before edges they establish are visible and nothing is left to
suppress
- suppressions were blunt (a race: entry matches any frame in the stack),
so they could also mask real races passing through those frames
- build with the spack gcc toolchain like every other job: no clang++, no
-fuse-ld=lld and no lld install step (the GNU BFD failure on tsan objects
was specific to clang's tsan runtime)
- use gcc 15 for the freshest libtsan runtime; the asan entry stays on
gcc 14, so the matrix now carries per-entry gcc and spack env names
- consume the tsan spack env and load the instrumented libstdc++ into each
test's environment via FAIRMQ_TEST_LD_LIBRARY_PATH (it shares the soname
of the compiler's own, so it substitutes process-wide at load time)
- use -fno-omit-frame-pointer for readable reports; optimization comes
from the project's Debug -Og
- verify the wiring: assert the test environment resolves libstdc++ to the
instrumented copy and that libzmq is tsan-instrumented, since both
failure modes are silent (the suite still passes, with reduced race
coverage)
- libzmq is not tsan-instrumented, so tsan cannot see the happens-before
its queues establish between user threads and libzmq I/O threads,
producing false-positive data races on message buffers
- add test/thread_sanitizer_suppressions.txt and point TSAN_OPTIONS at it
via the sanitizers job env so it reaches the tests and their device
subprocesses
- suppress: accesses made directly from libzmq, the zero-copy message
deleters libzmq runs from msg_t::close, shmem receive-side metadata
reads, and std::regex/locale lazy-init races in libstdc++
- the gate did `grep -q warning: build.log`, but build.log was never
produced by the cmake-action build, so under `set -e` the grep in the
`if` condition just reported "no match" and the job always passed
- as a result ~4961 clang-tidy warnings were silently ignored
- build manually and capture output to build.log with pipefail, and
fail explicitly if the log is missing or contains a warning
- tsan build failed at link with GNU ld:
"failed to set dynamic section sizes: bad value" (known binutils +
ThreadSanitizer incompatibility); install lld and select it via
-fuse-ld=lld for the tsan job only
- pass -fuse-ld=lld through cxx-flags so it reaches the link line,
avoiding the semicolon-list pitfall of
list(APPEND CMAKE_EXE_LINKER_FLAGS ...)
- build the bundled googletest with CMAKE_POSITION_INDEPENDENT_CODE=ON:
lld rejects R_X86_64_32 relocations from the non-PIC libgtest.a when
producing the position-independent tsan executable; the bundle is built
by a separate cmake invocation, so the flag must be set there
- Region.PreallocateInsideSession.shmem and Example.region.shmem failed
because mlock() of the managed segment/region hit RLIMIT_MEMLOCK on the
runner ("Cannot allocate memory"); the region example then hung until
its 30s timeout
- raise the limit via `sudo prlimit` in the same shell that launches
ctest (per-process, so it must be done here, not in a prior step)
- replace threeal/ctest-action with the equivalent ctest invocation
FairMQ's own sources (library, examples, tests) were recompiled from scratch
in every matrix job on every push.
- add hendrikmuhs/ccache-action to build, sanitizers and static-analysis jobs
- set CMAKE_C/CXX_COMPILER_LAUNCHER=ccache so cmake routes through it
- key the cache per (job, env, gcc) since ccache hashes the compiler
gcc was built from source (~58 min/job) because the FairMQ buildcache mirror
is only configured inside the env yaml, while gcc is installed before the env
is created.
- register the mirror globally after spack setup so "Install GCC" pulls the
compiler as a binary
- pin runners to ubuntu-24.04 so the weekly buildcache and weekday CI share an
image and concretize to matching hashes
- bump setup-spack to v3 to match the update-index job
