On my quest to generate reproducible standalone binaries for GNU FreeDink, I met new friends but currently lie defeated by an unexpected enemy...
- compiler version needs to be identical and recorded
- build options and their order need to be identical and recorder
- build path needs to be identical and recorded (otherwise debug symbols - and BuildIDs - change)
- diffoscope helps checking for differences in build output
-Wl,--no-insert-timestampfor .exe (with old binutils 2.25 caveat)
- no need to set a build path for stripped .exe (no ELF BuildID)
- reprotest helps checking build variations automatically
- MXE stack is apparently deterministic enough for a reproducible static build
- umask needs to be identical and recorded
- file timestamps needs to be set and recorded (more on this in a future episode)
First, the random build differences when using
-Wl,--no-insert-timestamp were explained.
peanalysis shows random build dates:
$ reprotest 'i686-w64-mingw32.static-gcc hello.c -I /opt/mxe/usr/i686-w64-mingw32.static/include -I/opt/mxe/usr/i686-w64-mingw32.static/include/SDL2 -L/opt/mxe/usr/i686-w64-mingw32.static/lib -lmingw32 -Dmain=SDL_main -lSDL2main -lSDL2 -lSDL2main -Wl,--no-insert-timestamp -luser32 -lgdi32 -lwinmm -limm32 -lole32 -loleaut32 -lshell32 -lversion -o hello && chmod 700 hello && analysePE.py hello | tee /tmp/hello.log-$(date +%s); sleep 1' 'hello' $ diff -au /tmp/hello.log-1* --- /tmp/hello.log-1490950327 2017-03-31 10:52:07.788616930 +0200 +++ /tmp/hello.log-1523203509 2017-03-31 10:52:09.064633539 +0200 @@ -18,7 +18,7 @@ found PE header (size: 20) machine: i386 number of sections: 17 - timedatestamp: -1198218512 (Tue Jan 12 05:31:28 1932) + timedatestamp: 632430928 (Tue Jan 16 09:15:28 1990) pointer to symbol table: 4593152 (0x461600) number of symbols: 11581 (0x2d3d) size of optional header: 224 @@ -47,7 +47,7 @@ Win32VersionValue: 0 size of image (memory): 4640768 size of headers (offset to first section raw data): 1536 - checksum (for drivers): 4927867 + checksum (for drivers): 4922616 subsystem: 3 win32 console binary DllCharacteristics: 0
These patches fix the variation and were submitted to MXE (pull request).
Next was playing with compiler support for SOURCE_DATE_EPOCH (which e.g. sets
The FreeDink DFArc frontend historically displays a build date in the About box:
"Build Date: %s\n", ..., __TDATE__
sadly support is only landing upstream in GCC 7 :/
I had to remove that date.
Now comes the challenging parts.
All my tests with reprotest checked. I started writing a reproducible build environment based on Docker (git browse).
At first I could not run reprotest in the container, so I reworked it with SSH support, and reprotest validated determinism.
(I also generate a reproducible .zip archive, more on that later.)
So far so good, but were the release identical when running reprotest successively on the different environments?
(reminder: this is a .exe build that is insensitive to varying path, hence consistent in a full reprotest)
$ sha256sum *.zip 189d0ca5240374896c6ecc6dfcca00905ae60797ab48abce2162fa36568e7cf1 freedink-109.0-bin-buildsh.zip e182406b4f4d7c3a4d239eee126134ba5c0304bbaa4af3de15fd4f8bda5634a9 freedink-109.0-bin-docker.zip e182406b4f4d7c3a4d239eee126134ba5c0304bbaa4af3de15fd4f8bda5634a9 freedink-109.0-bin-reprotest-docker.zip 37007f6ee043d9479d8c48ea0a861ae1d79fb234cd05920a25bb3db704828ece freedink-109.0-bin-reprotest-null.zip
Ouch! Even though both the Docker and my host are running Stretch, there are differences.
For the two host builds (direct and reprotest), there is a subtle but simple difference: HOME.
HOME is invariably non-existant in reprotest, while my normal compilation environment has an existing home (duh!).
This caused a subtle bug when cross-compiling with mingw and wine-binfmt:
- existing home:
./configureattempts to run conftest.exe, wine can create
~/.wine, conftest.exe runs with binfmt emulation, configure assumes:
checking whether we are cross compiling... no
- non-existing home:
./configureattempts to run
conftest.exe, wine can't create
~/.wine, conftest.exe fails, configure assumes:
checking whether we are cross compiling... yes
The respective binaries were very different notably due to a different
This can be fixed by specifying
--build in addition to
--host when calling
reprotest have one of the tests with a valid HOME (#860428).
Now comes the big one, after the fix I still got:
$ sha256sum *.zip 3545270ef6eaa997640cb62d66dc610a328ce0e7d412f24c8f18fdc7445907fd freedink-109.0-bin-buildsh.zip cc50ec1a38598d143650bdff66904438f0f5c1d3e2bea0219b749be2dcd2c3eb freedink-109.0-bin-docker.zip 3545270ef6eaa997640cb62d66dc610a328ce0e7d412f24c8f18fdc7445907fd freedink-109.0-bin-reprotest-chroot.zip cc50ec1a38598d143650bdff66904438f0f5c1d3e2bea0219b749be2dcd2c3eb freedink-109.0-bin-reprotest-docker.zip 3545270ef6eaa997640cb62d66dc610a328ce0e7d412f24c8f18fdc7445907fd freedink-109.0-bin-reprotest-null.zip
There is consistency on my host, and consistency within docker, but both are different.
Moreover, all the .o files were identical, so something must have gone wrong when compiling the libs, that is MXE.
After many checks it appears that
libstdc++.a is different.
Just overwriting it gets me a consistent FreeDink release on all environments.
Still, when rebuilding it (
libstdc++.a always has the same environment-dependent checksum.
45f8c5d50a68aa9919ee3602a4e3f5b2bd0333bc8d781d7852b2b6121c8ba27b /opt/mxe/usr/lib/gcc/i686-w64-mingw32.static/5.4.0/libstdc++.a # host 6870b84f8e17aec4b5cf23cfe9c2e87e40d9cf59772a92707152361b6ebc1eb4 /opt/mxe/usr/lib/gcc/i686-w64-mingw32.static/5.4.0/libstdc++.a # docker
The 2 libraries are much different, there's barely any blue in
Well before jumping to conclusion let's mix & match.
- First I
rsynca copy of my Docker filesystem and run it in a host chroot with a reset environment.
$ sudo env -i /usr/sbin/chroot chroot-docker/ $ exec bash -l $ cd /opt/mxe $ touch src/gcc.mk $ sha256sum /opt/mxe/usr/lib/gcc/i686-w64-mingw32.static/5.4.0/libstdc++.a 6870b84f8e17aec4b5cf23cfe9c2e87e40d9cf59772a92707152361b6ebc1eb4 /opt/mxe/usr/lib/gcc/i686-w64-mingw32.static/5.4.0/libstdc++.a $ make gcc [build] gcc i686-w64-mingw32.static [done] gcc i686-w64-mingw32.static 2709464 KiB 7m2.039s $ sha256sum /opt/mxe/usr/lib/gcc/i686-w64-mingw32.static/5.4.0/libstdc++.a 45f8c5d50a68aa9919ee3602a4e3f5b2bd0333bc8d781d7852b2b6121c8ba27b /opt/mxe/usr/lib/gcc/i686-w64-mingw32.static/5.4.0/libstdc++.a # consistent with host builds
- Then I import my previous reprotest chroot (plain debootstrap) in Docker:
$ sudo tar -C chroot -c . | docker import - chroot-debootstrap $ docker run -ti chroot-debootstrap /bin/bash $ sha256sum /opt/mxe/usr/lib/gcc/i686-w64-mingw32.static/5.4.0/libstdc++.a 45f8c5d50a68aa9919ee3602a4e3f5b2bd0333bc8d781d7852b2b6121c8ba27b /opt/mxe/usr/lib/gcc/i686-w64-mingw32.static/5.4.0/libstdc++.a $ touch src/gcc.mk $ make gcc [build] gcc i686-w64-mingw32.static [done] gcc i686-w64-mingw32.static 2709412 KiB 7m6.608s $ sha256sum /opt/mxe/usr/lib/gcc/i686-w64-mingw32.static/5.4.0/libstdc++.a 6870b84f8e17aec4b5cf23cfe9c2e87e40d9cf59772a92707152361b6ebc1eb4 /opt/mxe/usr/lib/gcc/i686-w64-mingw32.static/5.4.0/libstdc++.a # consistent with docker builds
So, AFAICS when building with:
- exactly the same kernel
- exactly the same GCC sources
- exactly the same host binaries
then depending on whether running in a container or not we get a consistent but different
This kind of issue is not detected with a simple
reprotest build, as it only tests variations within a fixed build environment.
This is quite worrisome, I intend to use a container to control my build environment, but I can't guarantee that the container technology will be exactly the same 5 years from now.
All my setup is simple and available for inspection at https://git.savannah.gnu.org/cgit/freedink.git/tree/autobuild/freedink-w32-snapshot/.
I'd very much welcome enlightenment