Let's review what we learned so far:
- compiler version need to be identical and recorded
- build options and their order needs to be identical and recorder
- build path needs to be identical and recorded
(otherwise debug symbols - and BuildIDs - change) - diffoscope helps checking for differences in build output
We stopped when compiling a PE .exe produced a varying output.
It turns out that PE carries a build date timestamp.
The spec says that bound DLLs timestamps are refered to in the "Delay-Load Directory Table". Maybe that's also the date Windows displays when a system-wide DLL is about to be replaced, too.
Build timestamps looks unused in .exe files though.
Anyway, Stephen Kitt pointed out (thanks!) that Debian's MinGW linker binutils-mingw-w64 has an upstream-pending patch that sets the timestamp to SOURCE_DATE_EPOCH if set.
Alternatively, one can pass -Wl,--no-insert-timestamp to set it to 0 (though see caveats below):
$ i686-w64-mingw32.static-gcc -Wl,--no-insert-timestamp hello.c -o hello.exe
$ md5sum hello.exe
298f98d74e6e913628a8b74514eddcb2 hello.exe
$ /opt/mxe/usr/bin/i686-w64-mingw32.static-gcc -Wl,--no-insert-timestamp hello.c -o hello.exe
$ md5sum hello.exe
298f98d74e6e913628a8b74514eddcb2 hello.exe
If we don't care about debug symbols, unlike with ELF, stripped PE binaries look stable too!
$ cd repro/
$ i686-w64-mingw32.static-gcc hello.c -o hello.exe && i686-w64-mingw32.static-strip hello.exe
$ md5sum hello.exe
6e07736bf8a59e5397c16e799699168d hello.exe
$ i686-w64-mingw32.static-gcc hello.c -o hello.exe && i686-w64-mingw32.static-strip hello.exe
$ md5sum hello.exe
6e07736bf8a59e5397c16e799699168d hello.exe
$ cd ..
$ cp -a repro repro2/
$ cd repro2/
$ i686-w64-mingw32.static-gcc hello.c -o hello.exe && i686-w64-mingw32.static-strip hello.exe
$ md5sum hello.exe
6e07736bf8a59e5397c16e799699168d hello.exe
Now that we have the main executable covered, what about the dependencies?
Let's see how well MXE compiles SDL2:
$ cd /opt/mxe/
$ cp -a ./usr/i686-w64-mingw32.static/lib/libSDL2.a /tmp
$ rm -rf * && git checkout .
$ make sdl2
$ md5sum ./usr/i686-w64-mingw32.static/lib/libSDL2.a /tmp/libSDL2.a
68909ab13181b1283bd1970a56d41482 ./usr/i686-w64-mingw32.static/lib/libSDL2.a
68909ab13181b1283bd1970a56d41482 /tmp/libSDL2.a
Neat - what about another build directory?
$ cd /usr/srx/mxe
$ make sdl2
$ md5sum usr/i686-w64-mingw32.static/lib/libSDL2.a /tmp/libSDL2.a
c6c368323927e2ae7adab7ee2a7223e9 usr/i686-w64-mingw32.static/lib/libSDL2.a
68909ab13181b1283bd1970a56d41482 /tmp/libSDL2.a
$ ls -l ./usr/i686-w64-mingw32.static/lib/libSDL2.a /tmp/libSDL2.a
-rw-r--r-- 1 me me 5861536 mars 23 21:04 /tmp/libSDL2.a
-rw-r--r-- 1 me me 5862488 mars 25 19:46 ./usr/i686-w64-mingw32.static/lib/libSDL2.a
Well that was expected.
But what about the filesystem order?
With such an automated build, could potential variations in the order of files go undetected?
Would the output be different on another filesystem format (ext4 vs. btrfs...)?
It was a good opportunity to test the disorderfs fuse-based tool.
And while I'm at it, check if reprotest is easy enough to use (the manpage is scary).
Let's redo our basic tests with it - basic usage is actually very simple:
$ apt-get install reprotest disorderfs faketime
$ reprotest 'make hello' 'hello'
...
will vary: environment
will vary: fileordering
will vary: home
will vary: kernel
will vary: locales
will vary: exec_path
will vary: time
will vary: timezone
will vary: umask
...
--- /tmp/tmpk5uipdle/control_artifact/
+++ /tmp/tmpk5uipdle/experiment_artifact/
│ --- /tmp/tmpk5uipdle/control_artifact/hello
├── +++ /tmp/tmpk5uipdle/experiment_artifact/hello
├── stat {}
│ │ @@ -1,8 +1,8 @@
│ │
│ │ Size: 8632 Blocks: 24 IO Block: 4096 regular file
│ │ Links: 1
│ │ -Access: (0755/-rwxr-xr-x) Uid: ( 1000/ me) Gid: ( 1000/ me)
│ │ +Access: (0775/-rwxrwxr-x) Uid: ( 1000/ me) Gid: ( 1000/ me)
│ │
│ │ Modify: 1970-01-01 00:00:00.000000000 +0000
│ │
│ │ Birth: -
# => OK except for permissions
$ reprotest 'make hello && chmod 755 hello' 'hello'
=======================
Reproduction successful
=======================
No differences in hello
c8f63b73265e69ab3b9d44dcee0ef1d2815cdf71df3c59635a2770e21cf462ec hello
$ reprotest 'make hello CFLAGS="-g -O2"' 'hello'
# => lots of differences, as expected
Now let's apply to the MXE build.
We keep the same build path, and also avoid using linux32 (because MXE would then recompile all the host compiler tools for 32-bit):
$ reprotest --dont-vary build_path,kernel 'touch src/sdl2.mk && make sdl2 && cp -a usr/i686-w64-mingw32.static/lib/libSDL2.a .' 'libSDL2.a'
=======================
Reproduction successful
=======================
No differences in libSDL2.a
d9a39785fbeee5a3ac278be489ac7bf3b99b5f1f7f3e27ebf3f8c60fe25086b5 libSDL2.a
That checks!
What about a full MXE environment?
$ reprotest --dont-vary build_path,kernel 'make clean && make sdl2 sdl2_gfx sdl2_image sdl2_mixer sdl2_ttf libzip gettext nsis' 'usr'
# => changes in installation dates
# => timestamps in .exe files (dbus, ...)
# => libicu doesn't look reproducible (derb.exe, genbrk.exe, genccode.exe...)
# => apparently ar timestamp variations in libaclui
Most libraries look reproducible enough.
ar differences may go away at FreeDink link time since I'm aiming at a static build.
Let's try!
First let's see how FreeDink behaves with stable dependencies.
We can compile with -Wl,--no-insert-timestamp and strip the binaries in a first step.
There are various issues (timestamps, permissions) but first let's check the executables themselves:
$ cd freedink/
$ reprotest --dont-vary build_path 'mkdir cross-woe-32/ && cd cross-woe-32/ && export PATH=/opt/mxe/usr/bin:$PATH && LDFLAGS='-Wl,--no-insert-timestamp' ../configure --host=i686-w64-mingw32.static --enable-static && make -j$(nproc) && make install-strip DESTDIR=$(pwd)/destdir' 'cross-woe-32/destdir/usr/local/bin'
# => executables are identical!
# Same again, just to make sure
$ reprotest --dont-vary build_path 'mkdir cross-woe-32/ && cd cross-woe-32/ && export PATH=/opt/mxe/usr/bin:$PATH && LDFLAGS='-Wl,--no-insert-timestamp' ../configure --host=i686-w64-mingw32.static --enable-static && make -j$(nproc) && make install-strip DESTDIR=$(pwd)/destdir' 'cross-woe-32/destdir/usr/local/bin'
│ --- /tmp/tmp2yw0sn4_/control_artifact/bin/freedink.exe
├── +++ /tmp/tmp2yw0sn4_/experiment_artifact/bin/freedink.exe
│ │ @@ -2,20 +2,20 @@
│ │ 00000010: b800 0000 0000 0000 4000 0000 0000 0000 ........@.......
│ │ 00000020: 0000 0000 0000 0000 0000 0000 0000 0000 ................
│ │ 00000030: 0000 0000 0000 0000 0000 0000 8000 0000 ................
│ │ 00000040: 0e1f ba0e 00b4 09cd 21b8 014c cd21 5468 ........!..L.!Th
│ │ 00000050: 6973 2070 726f 6772 616d 2063 616e 6e6f is program canno
│ │ 00000060: 7420 6265 2072 756e 2069 6e20 444f 5320 t be run in DOS
│ │ 00000070: 6d6f 6465 2e0d 0d0a 2400 0000 0000 0000 mode....$.......
│ │ -00000080: 5045 0000 4c01 0a00 e534 0735 0000 0000 PE..L....4.5....
│ │ +00000080: 5045 0000 4c01 0a00 0000 0000 0000 0000 PE..L...........
│ │ 00000090: 0000 0000 e000 0e03 0b01 0219 00f2 3400 ..............4.
│ │ 000000a0: 0022 4e00 0050 3b00 c014 0000 0010 0000 ."N..P;.........
│ │ 000000b0: 0010 3500 0000 4000 0010 0000 0002 0000 ..5...@.........
│ │ 000000c0: 0400 0000 0100 0000 0400 0000 0000 0000 ................
│ │ -000000d0: 00e0 8900 0004 0000 7662 4e00 0200 0000 ........vbN.....
│ │ +000000d0: 00e0 8900 0004 0000 89f8 4e00 0200 0000 ..........N.....
│ │ 000000e0: 0000 2000 0010 0000 0000 1000 0010 0000 .. .............
│ │ 000000f0: 0000 0000 1000 0000 00a0 8700 b552 0000 .............R..
│ │ 00000100: 0000 8800 d02d 0000 0050 8800 5006 0000 .....-...P..P...
│ │ 00000110: 0000 0000 0000 0000 0000 0000 0000 0000 ................
│ │ 00000120: 0060 8800 4477 0100 0000 0000 0000 0000 .`..Dw..........
│ │ 00000130: 0000 0000 0000 0000 0000 0000 0000 0000 ................
│ │ 00000140: 0440 8800 1800 0000 0000 0000 0000 0000 .@..............
├── stat {}
│ │ │ @@ -1,8 +1,8 @@
│ │ │
│ │ │ Size: 5121536 Blocks: 10008 IO Block: 4096 regular file
│ │ │ Links: 1
│ │ │ Access: (0755/-rwxr-xr-x) Uid: ( 1000/ me) Gid: ( 1000/ me)
│ │ │
│ │ │ -Modify: 2017-03-26 01:26:35.233841833 +0000
│ │ │ +Modify: 2017-03-26 01:27:01.829592505 +0000
│ │ │
│ │ │ Birth: -
Gah...
AFAIU there is something random in the linking phase, and sometimes the timestamp is removed, sometimes it's not.
Not very easy to track but I believe I reproduced it with the "hello" example:
# With MXE:
$ reprotest 'i686-w64-mingw32.static-gcc hello.c -I /opt/mxe/usr/i686-w64-mingw32.static/include -I/opt/mxe/usr/i686-w64-mingw32.static/include/SDL2 -L/opt/mxe/usr/i686-w64-mingw32.static/lib -lmingw32 -Dmain=SDL_main -lSDL2main -lSDL2 -lSDL2main -Wl,--no-insert-timestamp -luser32 -lgdi32 -lwinmm -limm32 -lole32 -loleaut32 -lshell32 -lversion -o hello && chmod 700 hello' 'hello'
# => different
# => maybe because it imports the build timestamp from -lSDL2main
# With Debian's MinGW (but without SOURCE_DATE_EPOCH):
$ reprotest 'i686-w64-mingw32-gcc hello.c -I /opt/mxe/usr/i686-w64-mingw32.static/include -I/opt/mxe/usr/i686-w64-mingw32.static/include/SDL2 -L/opt/mxe/usr/i686-w64-mingw32.static/lib -lmingw32 -Dmain=SDL_main -lSDL2main -lSDL2 -lSDL2main -Wl,--no-insert-timestamp -luser32 -lgdi32 -lwinmm -limm32 -lole32 -loleaut32 -lshell32 -lversion -o hello && chmod 700 hello' 'hello'
=======================
Reproduction successful
=======================
No differences in hello
0b2d99dc51e2ad68ad040d90405ed953a006c6e58599beb304f0c2164c7b83a2 hello
# Let's remove -Dmain=SDL_main and let our main() have precedence over the one in -lSDL2main:
$ reprotest 'i686-w64-mingw32.static-gcc hello.c -I /opt/mxe/usr/i686-w64-mingw32.static/include -I/opt/mxe/usr/i686-w64-mingw32.static/include/SDL2 -L/opt/mxe/usr/i686-w64-mingw32.static/lib -lmingw32 -lSDL2main -lSDL2 -lSDL2main -Wl,--no-insert-timestamp -luser32 -lgdi32 -lwinmm -limm32 -lole32 -loleaut32 -lshell32 -lversion -o hello && chmod 700 hello' 'hello'
=======================
Reproduction successful
=======================
No differences in hello
6c05f75eec1904d58be222cc83055d078b4c3be8b7f185c7d3a08b9a83a2ef8d hello
$ LANG=C i686-w64-mingw32.static-ld --version # MXE
GNU ld (GNU Binutils) 2.25.1
Copyright (C) 2014 Free Software Foundation, Inc.
$ LANG=C i686-w64-mingw32-ld --version # Debian
GNU ld (GNU Binutils) 2.27.90.20161231
Copyright (C) 2016 Free Software Foundation, Inc.
It looks like there is a random behavior in binutils 2.25, coupled with SDL2's wrapping of my main().
So FreeDink is nearly reproducible, except for this build timestamp issue that pops up in all kind of situations. In the worse case I can zero it out, or patch MXE's binutils until they upgrade.
More importantly, what if I recompile FreeDink and the dependencies twice?
$ (cd /opt/mxe/ && make clean && make sdl2 sdl2_gfx sdl2_image sdl2_mixer sdl2_ttf glm libzip gettext nsis)
$ (mkdir cross-woe-32/ && cd cross-woe-32/ \
&& export PATH=/opt/mxe/usr/bin:$PATH \
&& LDFLAGS="-Wl,--no-insert-timestamp" ../configure --host=i686-w64-mingw32.static --enable-static \
&& make V=1 -j$(nproc) \
&& make install-strip DESTDIR=$(pwd)/destdir)
$ mv cross-woe-32/ cross-woe-32-1/
# Same again...
$ mv cross-woe-32/ cross-woe-32-2/
$ diff -ru cross-woe-32-1/destdir/ cross-woe-32-2/destdir/
[nothing]
Yay!
I could not reproduce the build timestamp issue in the stripped binaries, though it was still varying in the unstripped src/freedinkedit.exe.
I mentioned there was other changes noticed by diffoscope.
- Changes in file timestamps.
That one is interesting.
Could be ignored, but we want to generate an identical binary package/archive too, right?
That's where archive meta-data matters.
make INSTALL="$(which install) install -p" could help for static files, but not generated ones.
The doc suggests clamping all files to SOURCE_DATE_EPOCH - i.e. all generated files will have their date set at that timestamp:
$ export SOURCE_DATE_EPOCH=$(date +%s) \
&& reprotest --dont-vary build_path \
'make ... && find destdir/ -newermt "@${SOURCE_DATE_EPOCH}" -print0 | xargs -0r touch --no-dereference --date="@${SOURCE_DATE_EPOCH}"' 'cross-woe-32/destdir/'
- Changes in directory permissions
Caused by varying umask.
I attempted to mitigate the issue by playing with make install MKDIR_P="mkdir -p -m 755" (1).
However even mkdir -p -m ... does not set permissions for intermediate directories.
Maybe it's better to set and record the umask...
So, aside from minor issues such as BuildIDs and build timestamps, the toolchain is pretty stable as of now.
The issue is more about fixing and recording the build environment.
Which is probably the next challenge 