RSS Atom Add a new post titled:

I'm mirroring and reworking a large Git repository with git filter-branch (conversion ETA: 20h), and I was wondering how to use --state-branch which is supposed to speed-up later updates, or split a large conversion in several updates.

The documentation is pretty terse, the option can produce weird results (like an identity mapping that breaks all later updates, or calling the expensive tree-filter but discarding the results), wrappers are convoluted, but I got something to work so I'll share :)

The main point is: run the initial script and the later updates in the same configuration, which means the target branch needs to be reset to the upstream branch each time, before it's rewritten again by filter-branch. In other words, don't re-run it on the rewritten branch, nor attempt some complex merge/cherry-pick.

git fetch
git branch --no-track -f myrewrite origin/master
git filter-branch \
  --xxx-filter ... \
  --xxx-filter ... \
  --state-branch refs/heads/filter-branch/myrewrite \
  -d /dev/shm/filter-branch/$$ -f \
  myrewrite

Updates restart from scratch but only take a few seconds to skim through all the already-rewritten commits, and maintain a stable history.

Note that if the process is interrupted, the state-branch isn't modified, so it's not a stop/resume feature. If you want to split a lenghty conversion, you could simulate multiple upstream updates by checking out successive points in history (e.g. per year using $(git rev-list -1 --before='2020-01-01 00:00:00Z')).

--state-branch isn't meant to rewrite in reverse chronological order either, because all commit ids would constantly change. Still, you can rewrite only the recent history for a quick discardable test.

Be cautious when using/deleting rewritten branches, especially during early tests, because Git tends to save them to multiple places which may desync (e.g. .git/refs/heads/, .git/logs/refs/, .git/packed-refs). Also remember to delete the state-branch between different tests. Last, note the unique temporary directory -d to avoid ruining concurrent tests ^_^'

Posted Sun Oct 4 10:18:57 2020 Tags:

Debian LTS Logo

Here is my transparent report for my work on the Debian Long Term Support (LTS) and Debian Extended Long Term Support (ELTS), which extend the security support for past Debian releases, as a paid contributor.

In September, the monthly sponsored hours were split evenly among contributors depending on their max availability - I was assigned 19.75h for LTS (out of my 30 max; all done) and 20h for ELTS (out of my 20 max; all done).

ELTS - Jessie

  • qemu: jessie triage: finish work started in August
  • qemu: backport 5 CVE fixes, perform virtual and physical testing, security upload ELA-283-1
  • libdbi-perl: global triage: clarifications, confirm incomplete and attempt to get upstream action, request new CVE following discussion with security team
  • libdbi-perl: backport 5 CVE fixes, test, security upload ELA-285-1

LTS - Stretch

  • qemu: stretch triage, while working on ELTS update; mark several CVEs unaffected, update patch/status
  • wordpress: global triage: reference new patches, request proper CVE to fix our temporary tracking
  • wordpress: revamp package: upgrade to upstream's stable 4.7.5->4.7.18 to ease future updates, re-apply missing patches, fix past regression and notify maintainer, security upload DLA-2371-1
  • libdbi-perl: common work with ELTS, security upload DLA-2386-1
  • public IRC team meeting

Documentation/Scripts

  • LTS/TestSuites/wordpress: new page with testsuite import and manual tests
  • LTS/TestSuites/qemu: minor update
  • wiki.d.o/Sympa: update Sympa while using it as a libdbi-perl reverse-dep test (update for newer versions, explain how to bootstrap admin access)
  • www.d.o/lts/security: import a couple missing announcements and notify uploaders about procedures
  • Check status for pdns-recursor, following user request
  • Check status for golang-1.7 / CVE-2019-9514 / CVE-2019-9512
  • Attempt to improve cooperation after seeing my work discarded and redone as-is, which sadly isn't the first time; no answer
  • Historical analysis of our CVE fixes: experiment to gather per-CVE tracker history
Posted Thu Oct 1 16:08:50 2020 Tags:

Debian LTS Logo

Here is my transparent report for my work on the Debian Long Term Support (LTS) and Debian Extended Long Term Support (ELTS), which extend the security support for past Debian releases, as a paid contributor.

In August, the monthly sponsored hours were split evenly among contributors depending on their max availability - I was assigned 21.75h for LTS (out of my 30 max; all done) and 14.25h for ELTS (out of my 20 max; all done).

We had a Birds of a Feather videoconf session at DebConf20, sadly with varying quality for participants (from very good to unusable), where we shared the first results of the LTS survey.

There were also discussions about evaluating our security reactivity, which proved surprisingly hard to estimate (neither CVE release date and criticality metrics are accurate nor easily available), and about when it is appropriate to use public naming in procedures.

Interestingly ELTS gained new supported packages, thanks to a new sponsor -- so far I'd seen the opposite, because we were close to the EOL.

As always, there were opportunities to de-dup work through mutual cooperation with the Debian Security team, and LTS/ELTS similar updates.

ELTS - Jessie

  • Fresh build VMs
  • rails/redmine: investigate issue, initially no-action as it can't be reproduced on Stretch and isn't supported in Jessie; follow-up when it's supported again
  • ghostscript: global triage: identify upstream fixed version, distinguish CVEs fixed within a single patch, bisect non-reproducible CVEs, reference missing commit (including at MITRE)
  • ghostscript: fix 25 CVEs, security upload ELA-262-1
  • ghostscript: cross-check against the later DSA-4748-1 (almost identical)
  • software-properties: jessie triage: mark back for update, at least for consistency with Debian Stretch and Ubuntu (all suites)
  • software-properties: security upload ELA-266-1
  • qemu: global triage: update status and patch/regression/reproducer links for 6 pending CVEs
  • qemu: jessie triage: fix 4 'unknown' lines for qemu following changes in package attribution for XSA-297, work continue in September

LTS - Stretch

  • sane-backends: global triage: sort and link patches for 7 CVEs
  • sane-backends: fix dep-8 test and notify the maintainer,
  • sane-backends: security upload DLA-2332-1
  • ghostscript: security upload DLA 2335-1 (cf. common ELTS work)
  • ghostscript: rebuild ("give back") on armhf, blame armhf, get told it was a concurrency / build system issue -_-'
  • software-properties: security upload DLA 2339-1 (cf. common ELTS work)
  • wordpress: global triage: reference regression for CVE-2020-4050
  • wordpress: stretch triage: update past CVE status, work continues in September with probably an upstream upgrade 4.7.5 -> 4.7.18
  • nginx: cross-check my July update against the later DSA-4750-1 (same fix)
  • DebConf BoF + IRC follow-up

Documentation/Scripts

Posted Tue Sep 1 10:48:31 2020 Tags:

planet.gnu.org logo

The system running planet.gnu.org was upgraded/reinstalled to Debian 10 "buster" :)
Documentation was updated.

Let me know if you notice any issue - planet@gnu.org.

For the next upgrade, we'll have to decide whether to takeover Planet Venus and upgrade it to Python 3, or migrate to another Planet software.
Suggestions/help welcome :)

Posted Sat Aug 15 06:55:30 2020 Tags:

Debian LTS Logo

Here is my transparent report for my work on the Debian Long Term Support (LTS) and Debian Extended Long Term Support (ELTS), which extend the security support for past Debian releases, as a paid contributor.

In July, the monthly sponsored hours were split evenly among contributors depending on their max availability - I was assigned 25.25h for LTS (out of 30 max; all done) and 13.25h for ELTS (out of 20 max; all done).

We shifted suites: welcome Stretch LTS and Jessie ELTS. The LTS->ELTS switch happened at the start of the month, but the oldstable->LTS switch happened later (after finalizing and flushing proposed-updates to a last point release), causing some confusion but nothing major.

ELTS - Jessie

  • New local build setup
  • ELTS buildds: request timezone harmonization
  • Reclassify in-progress updates from jessie-LTS to jessie-ELTS
  • python3.4: finish preparing update, security upload ELA 239-1
  • net-snmp: global triage: bisect CVE-2019-20892 to identify affected version, jessie/stretch not-affected
  • nginx: global triage: clarify CVE-2013-0337 status; locate CVE-2020-11724 original patch and regression tests, update MITRE
  • nginx: security upload ELA-247-1 with 2 CVEs

LTS - Stretch

  • Reclassify in-progress/needed updates from stretch/oldstable to stretch-LTS
  • rails: upstream security: follow-up on CVE-2020-8163 (RCE) on upstream bug tracker and create pull request for 4.x (merged), hence getting some upstream review
  • rails: global security: continue coordinating upload in multiple Debian versions, prepare fixes for common stretch/buster vulnerabilities in buster
  • rails: security upload DLA-2282 fixing 3 CVEs
  • python3.5: security upload DLA-2280-1 fixing 13 pending non-critical vulnerabilities, and its test suite
  • nginx: security upload DLA-2283 (cf. common ELTS work)
  • net-snmp: global triage (cf. common ELTS work)
  • public IRC monthly team meeting
  • reach out to clarify the intro from last month's report, following unsettled feedback during meeting

Documentation/Scripts

  • ELTS/README.how-to-release-an-update: fix typo
  • ELTS buildd: attempt to diagnose slow perfs, provide comparison with Debian and local builds
  • LTS/Meetings: improve presentation
  • SourceOnlyUpload: clarify/de-dup pbuilder doc
  • LTS/Development: reference build logs URL, reference proposed-updates issue during dists switch, reference new-upstream-versioning discussion, multiple jessie->stretch fixes and clean-ups
  • LTS/Development/Asan: drop wheezy documentation
  • Warn about jruby mis-triage
  • Provide feedback for ksh/CVE-2019-14868
  • Provide feedback for condor update
  • LTS/TestsSuites/nginx: test with new request smuggling test cases
Posted Mon Aug 3 13:52:10 2020 Tags:

Debian LTS Logo

Here is my transparent report for my work on the Debian Long Term Support (LTS) and Debian Extended Long Term Support (ELTS), which extend the security support for past Debian releases, as a paid contributor.

In June, the monthly sponsored hours were split evenly among contributors depending on their max availability - I was assigned 30h for LTS (out of 30 max; all done) and 5.25h for ELTS (out of 20 max; all done).

While LTS is part of the Debian project, fellow contributors sometimes surprise me: suggestion to vote for sponsors-funded projects with concorcet was only met with overhead concerns, and there were requests for executive / business owner decisions (we're currently heading towards consultative vote); I heard concerns about discussing non-technical issues publicly (IRC team meetings are public though); the private mail infrastructure was moved from self-hosting straight to Google; when some got an issue with Debian Social for our first video conference, there were immediate suggestions to move to Zoom...
Well, we do need some people to make those LTS firmware updates in non-free :)

Also this was the last month before shifting suites: goodbye to Jessie LTS and Wheezy ELTS, welcome Stretch LTS and Jessie ELTS.

ELTS - Wheezy

  • mysql-connector-java: improve testsuite setup; prepare wheezy/jessie/stretch triple builds; coordinate versioning scheme with security-team; security upload ELA 234-1
  • ntp: wheezy+jessie triage: 1 ignored (too intrusive to backport); 1 postponed (hard to exploit, no patch)
  • Clean-up (ditch) wheezy VMs :)

LTS - Jessie

  • mysql-connector-java: see common work in ELTS
  • mysql-connector-java: security uploads DLA 2245-1 (LTS) and DSA 4703 (oldstable)
  • ntp: wheezy+jessie triage (see ELTS)
  • rails: global triage, backport 2 patches, security upload DLA 2251-1
  • rails: global security: prepare stretch/oldstable update
  • rails: new important CVE on unmaintained 4.x, fixes introduce several regressions, propose new fix to upstream, update stretch proposed update [and jessie, but rails will turn out unsupported in ELTS]
  • python3.4: prepare update to fix all pending non-criticial issues, 5/6 ready
  • private video^W^Wpublic IRC team meeting

Documentation/Scripts

Posted Wed Jul 1 14:19:14 2020 Tags:

Debian LTS Logo

Here is my transparent report for my work on the Debian Long Term Support (LTS) and Debian Extended Long Term Support (ELTS), which extend the security support for past Debian releases, as a paid contributor.

In May, the monthly sponsored hours were split evenly among contributors depending on their max availability - I was assigned 17.25h for LTS (out of 30 max; all done) and 9.25h for ELTS (out of 20 max; all done).

A survey will be published very shortly to gather feedback from all parties involved in LTS (users, other Debian teams...) -- let us know what you think, so we start the forthcoming new (Stretch) LTS cycle in the best conditions :)

Discussion is progressing on funding & governance of larger LTS-related projects. Who should decide: contributors, Freexian, sponsors? Do we fund with a percentage or by capping resources allocated on security updates? I voiced concerns over funding these at the expense of smaller, more organic, more recurrent tasks that are less easy to specify but greatly contribute to the overall quality nevertheless.

ELTS - Wheezy

  • mysql-connector-java: upgrade to 5.1.49, refresh patches, document/run test suite, prepare upload, prepare upgrade path (+ see LTS)
  • CVE-2020-3810/apt: triage (affected), enquire about failing test, run testsuite, security upload ELA 228-1

LTS - Jessie

  • ansible: global triage: finish last month's triage, fix affected versions, provide reproducer
  • ansible: backport patches to early version, security upload DLA 2202-1
  • mysql-connector-java: propose 5.1.49 update to all dists (+ see ELTS)
  • CVE-2019-20637/varnish: global triage: ping upstream, get PoC, determine status for all Debian dists, jessie not-affected
  • public IRC team meeting

Documentation/Scripts

Posted Tue Jun 2 11:29:02 2020 Tags:

Debian LTS Logo

Here is my transparent report for my work on the Debian Long Term Support (LTS) and Debian Extended Long Term Support (ELTS), which extend the security support for past Debian releases, as a paid contributor.

In April, the monthly sponsored hours were split evenly among contributors depending on their max availability - I was assigned 28.75h for LTS (out of 30 max; all done) and 7.75h for ELTS (out of 20 max; I did 2.75).

Escalation procedures were (internally) documented with a focus on discussing issues with team coordinator(s) first.

Debian LTS had its first team meeting through IRC and lots of workflow question were discussed. This should help discuss questions that are a bit hard to bring up, and ensure everybody participates. There were lots of topics and it was a bit rushed, but this is something we want to repeat monthly now, possibly with audio/video in a couple months.

Remarks from last month's report were discussed, strengthening the Front-Desk role.

10% of the global funding is now reserved for infrastructure work. What kind of work, and who (LTS or external) will do the work, will be discussed further.

A fellow DD suggested (in a private conversation) that LTS may be taking time from the Debian Security team, due to additional commits to review. Conversely, this is another opportunity to mention all the global, non-LTS-specific work that LTS provides, which I usually highlight in my reports, and maybe I should be even more ;)

ELTS - Wheezy

  • CVE-2020-11612/netty: triage: ignored (deceptively hard to backport, OOM mitigation only)
  • mysql-connector-java: triage: in-progress (subscription-only update from Oracle, attempt to find more detail, waiting for public version)
  • CVE-2020-11868/ntp: global triage: identify and reference missing patch, coordinate with uploader

LTS - Jessie

  • netty, mysql-connector-java, ntp: common triage (see above)
  • CVE-2019-20637/varnish: global triage: attempt to reproduce, attempt to get PoC/vulnerable versions from upstream, update BTS
  • ansible: jessie triage: reset ignore->no-dsa old vulnerabilites after discussing with initial triager
  • ansible: global triage: identify more affected version ranges, locate more patches
  • ansible: prepare jessie upload (work-in-progress)
  • tiff: suites harmonization: offer to work on a tiff/stretch update, follow-up on maintainer's questions, who eventually did the update
  • dsa-needed.txt: identify stale entries from inactive LTS contributor, check for status
  • team meeting: see minutes

Documentation/Scripts

Posted Sat May 2 11:25:57 2020 Tags:

Debian LTS Logo

Here is my transparent report for my work on the Debian Long Term Support (LTS) and Debian Extended Long Term Support (ELTS), which extend the security support for past Debian releases, as a paid contributor.

In March, the monthly sponsored hours were split evenly among contributors depending on their max availability - I was assigned 30h for LTS (out of 30 max; all done) and 20h for ELTS (out of 20 max; I did 0).

Most contributors claimed vulnerabilities by performing early CVE monitoring/triaging on their own, making me question the relevance of the Front-Desk role. It could be due to a transient combination of higher hours volume and lower open vulnerabilities.

Working as a collective of hourly paid freelancers makes it more likely to work in silos, resulting in little interaction when raising workflow topics on the mailing list. Maybe we're reaching a point where regular team meetings will be benefical.

As previously mentioned, I structure my work keeping the global Debian security in mind. It can be stressful though, and I believe current communication practices may deter such initiatives.

ELTS - Wheezy

  • No work. ELTS has few sponsors right now and few vulnerabilities to fix, hence why I could not work on it this month. I gave back my hours at the end of the month.

LTS - Jessie

  • lua-cgi: global triage: CVE-2014-10399,CVE-2014-10400/lua-cgi not-affected, CVE-2014-2875/lua-cgi referenced in BTS
  • libpcap: global triage: request CVE-2018-16301 rejection as upstream failed to; got MITRE to reject (not "dispute") a CVE for the first time!
  • nfs-utils: suites harmonization: CVE-2019-3689: ping upstream again, locate upstream'd commit, reference it at BTS and MITRE; close MR which had been ignored and now redone following said referencing
  • slurm-llnl: re-add; create CVE-2019-12838 reproducer, test abhijith's pending upload; reference patches; witness regression in CVE-2019-19728, get denied access to upstream bug, triage as ignored (minor issue + regression); security upload DLA 2143-1
  • xerces-c: global triage progress: investigate ABI-(in)compatibility of hle's patch direction; initiate discussion at upstream and RedHat; mark postponed
  • nethack: jessie triage fix: mark end-of-life
  • tor: global triage fix: CVE-2020-10592,CVE-2020-10593: fix upstream BTS links, fix DSA reference
  • php7.3: embedded copies: removed from unstable (replaced with php7.4); checked whether libonig is still bundled (no, now properly unbundled at upstream level); jessie still not-affected
  • okular: CVE-2020-9359: reference PoC, security upload DLA 2159-1

Documentation/Scripts

  • data/dla-needed.txt: tidy/refresh pending packages status
  • LTS/Development: DLA regression numbering when a past DLA affects a different package
  • LTS/FAQ: document past LTS releases archive location following a user request; trickier than expected, 3 contributors required to find the answer ;)
  • Question aggressive package claims; little feedback
  • embedded-copies: libvncserver: reference various state of embedded copies in italc/ssvnc/tightvnc/veyon/vncsnapshot; builds on initial research from sunweaver
  • Attempt to progress on libvncserver embedded copies triaging; technical topic not anwered, organizational topic ignored
  • phppgadmin: provide feedback on CVE-2019-10784
  • Answer general workflow question about vulnerability severity
  • Answer GPAC CVE information request from a PhD student at CEA, following my large security update
Posted Wed Apr 1 14:26:39 2020 Tags:

Debian LTS Logo

Here is my transparent report for my work on the Debian Long Term Support (LTS) and Debian Extended Long Term Support (ELTS), which extend the security support for past Debian releases, as a paid contributor.

In February, the monthly sponsored hours were split evenly among contributors depending on their max availability - I was assigned 20h for LTS (out of 30 max; all done) and 8h for ELTS (out of 20 max; I did 7).

Security work is never completely isolated, typically my work on nodejs impacted jessie/stretch/buster, and my work on netty affected wheezy/jessie/stretch :)

ELTS - Wheezy

  • netty: refine prior triages, write minimal test server, adapt 3 fixes, security upload: ELA-214
  • Suggest redispatching hours from past month not given back in time, as team members only got 3.5h each; follow-up on the issue
  • Contribute to exchanges about supporting libgd2 (unsupported dependency of a supported package, an inconsistency we'll try to detect earlier)

LTS - Jessie

  • netty: refine prior triages, security upload DLA 2109-1
  • netty-3.9: identify duplicate package, fix prior vulnerabilities, security upload DLA 2110-1
  • nodejs: jessie/stretch/buster triage (3 CVEs), request access to not-yet-public hackerone reports
  • nodejs: clarify support status, reclassify open vulnerabilities on nodejs ecosystem as EOL (end-of-life) for jessie & stretch
  • http-parser: mark as affected by nodejs' CVE-2019-15605; jessie triage: ignored (invasive change with ABI breakage)
  • wordpress: precise my past triage (2 CVEs): postponed (serialization vulnerabilities related to PHP itself currently not addressed at application/wordpress level)
  • otrs2: security upload DLA 2118-1 (interestingly recent otrs2 is in non-free not due to licensing, but due to embedding specific versions of javascript dependencies)
  • CVE-2019-10784/phppgadmin: answer request for comment
  • xen: point out external support

Documentation/Scripts

  • TestSuites/netty: instruction on how to find, compile and adapt server examples
  • DLA-1993-1: update Debian website (was only published via mailing-list)
  • embedded-code-copies: reference http-parser embedded in nodejs
  • README.external-support: clean-up external support contact points
Posted Mon Mar 2 10:30:55 2020 Tags:

This blog is powered by ikiwiki.