Release security fixes without additional maintenance fixes

Teams and Committees Core
1

Problem:
Whenever we have security releases, they also contain the bugfixes and maintenance work that also happen to have landed in the branch since the release before.

As an example here is the change log for the 10.4.18 security release: TYPO3 10.4.18 Release Notes - get.typo3.org

In a project with a high demand for stability this kind of release raises a conflict between

  • rolling out the security release to production as quickly as possible to mitigate any risks of attacks and
  • scanning all other included changes for possible implications on the projects and thorough retesting.

This causes stress all while you’re typically busy with other work that was planned before. If security and bugfix releases were separated, we could roll out the security fixes quicker and with more confidence and we could easily take our time for planning, examining and testing the maintenance releases.

Proposal:
The way security fixes are handled and only merged into the branches shortly before the release should be kept as it is, to ensure the confidentiality of security issues. But for the security release itself a security release branch should be created, that is based on the release before and only the security relevant fixes should be cherry picked into it. The security release is tagged on that branch.

That applies to every major version, that is supported at the time. Currently for a security release that would mean creating 3 security release branches 11.3.4-sec, 10.4.21-sec and 9.5.31-sec (i wouldn’t include master).

This is what the 10.4.18 release could have looked like in terms of git branching:
Screenshot 2021-08-17 at 10.28.20

Alternative:
In our website project we could do just that ourselves and fork TYPO3 on every security release and package just the security fixes, or include the security fixes via composer patches. But I think we are not the only project with that demand and the enterprise CMS TYPO3 could provide this.

2

Thanks for bringing up this topic. In previous years, “releasing from last Git tag” has been discussed a couple of times, however seemed to be incompatible with existing Gerrit workflow and would have lead to much more versions (which basically was the most important aspect not to do it).

Please don’t understand my remarks just as notes/input/ideas, they are not meant to stop this initiative per se.

Scenario

Last good releases were 11.3.0, 10.4.17, 9.5.27.

In general there will be people that are interested in

  • receiving all recent bug fixes or a branch (e.g. 10.4) and security fixes, e.g. upgrading from 10.4.17 to 10.4.18
  • receiving just the security fixes on top of the last good version, e.g. 10.4.17 to 10.4.17.1 (that’s the topic that I’m going to focus on)

Timeline

  • May 11th, 2021: last good releases
    • current LTS
    • branch 10.4, tag v10.4.17
    • branch 9.5, tag v9.5.27
  • July 13th, 2021: last good release
    • new upcoming LTS
    • branch master, tag v11.3.0
  • July 20th, 2021: security releases
    • new branch 11.3.0 from tag v11.3.0 (last good)
    • new branch 10.4.17 from tag v10.4.17 (last good)
    • new branch 9.5.27 from tag v9.5.27 (last good)
    • new tag v11.3.0.1 from branch 11.3.0 (just security fixes - on top of last good version)
    • new tag v10.4.18.0 from branch 10.4 (bug fixes + security fixes)
    • new tag v10.4.17.1 from branch 10.4.17 (just security fixes)
    • new tag v9.5.28.0 from branch 9.5 (bug fixes + security fixes)
    • new tag v9.5.27.1 from branch 9.5.27 (just security fixes - on top of last good version)
  • August 10th, 2021: security releases
    • turned out last releases contained regressions (not related to security topics)
    • new tag v11.3.0.2 from branch 11.3.0 (just security fixes)
    • new tag v10.4.19.0 from branch 10.4 (all bug fixes + security fixes - regression fixed)
    • new tag v10.4.17.2 from branch 10.4.17 (just security fixes - on top of last good version)
    • new tag v9.5.29.0 from branch 9.5 (bug fixes + security fixes - regression fixed)
    • new tag v9.5.27.1 from branch 9.5.27 (just security fixes - on top of last good version)
  • August 16th, 2021: maintenance releases
    • turned out last releases contained regressions (related to both security and non-security topics)
    • new tag v11.3.0.3 from branch 11.3.0 (just security fixes)
    • new tag v10.4.20.0 from branch 10.4 (all bug fixes + security fixes - regression fixed)
    • new tag v10.4.17.3 from branch 10.4.17 (just security fixes - on top of last good version)
    • new tag v9.5.29.0 from branch 9.5 (bug fixes + security fixes - regression fixed)
    • new tag v9.5.27.1 from branch 9.5.27 (just security fixes - on top of last good version)

Observation

  • security fixes in sprint releases like v11.3.0 use a “release from last good tag” strategy already
  • releasing from “last tag” actually implies to create a new branch from a particular patch level version (tag v10.4.17 → branch 10.4.17)
    • gap for not having “normal” bug fixes is around three months (May to August)
    • upgrading from e.g. v10.4.17.3 to (future) v10.4.21.0 might be disappointing - since “complete” releases are not used/tested anymore
  • in case there are regressions, the gap between last good and security fixes + bug fixes + regression fixes get bigger
  • amount of versions and tarball/zip bundles increases
    • 2 → 4 (no sprint-releases like 11.3 available)
    • 3 → 5 (sprint-releases available)

Challenge

  • chicken-egg-problem when people are not using recent versions (having bug fixes + security fixes) anymore - those combined releases would exist, but less people are using/testing them, potential regressions there are detected much later
  • in case a majority would just use those security releases - that don’t have regular bug fixes… is it worth to maintain bug fix versions at all and back-port bugs from master branch to previous branches?

Again: Before changing things like this, we need to understand the big picture and think about potential consequences. Personally I’m of course open to discuss that topic further.

3

I have long ago maintained some releases of TYPO3 and we had discussed this topic a couple of times in the team already in the past. Mostly these discussions started exactly after this kind of “situation”: some patch-level update “stressed” the community due to some regression.

Overall the result of these discussions had been that the benefits of “doing it” (separating the security from the maintenance fixes) did not really outweighed the drawbacks (a more complicated release branching model, scripts, and logistics around the release process) - as you can see with the outstanding overview/example @ohader provided above.

Think of it this way: the developers and reviewers of a bugfix for any patch level bugfix (regardless if its security relevant or not) are always concerned as much as possible about backwards compatibility, not breaking stuff and having a “smooth” update (no DB updates required, just install the code and “you are good”).

For several reasons, sometimes these guidelines are not gonna work out, and we will end up having a patch level release which contains either a regression or some security fix which requires more attention (i.e. special testing etc).

The risk of breaking in a patch level is very low but it’s always there, and it does not really matter if it is a security fix or another fix. In special circumstances, i.e. when many fixes have already dropped into the branch without a release, the maintainers can always choose to make a “maintenance” release and post-pone the security fixes for a week or so (currently not possible the other way around due to the “we release everything that is merged” strategy).

Even thou this kind of errors can happen, I still have lots of trust in the process, since I know the guys behind it very well and know how much attention to details and dedication they have (usually this kind of issues turn up to be “Ehrensache”) to solve it as quick as possible after a potential release with a regression.

You on your end need to care about every patchlevel release regardless if its security or not anyway. In my opinion, the security issues in the core have been very “low impact” lately (requiring backend access and at most allowing editors / admins to create XSS attacks), so that we usually do not require to install them “right away”, so that in this time you can also care about other fixes that might have been included.

I know lots of people just install the patch-level releases that are marked as “security” (so skipping the other maintenance release in between), so having a separate release just with security does not help in this case anyway.

Basically, I am against making it too complicated just for this very edge cases which happens once every couple of years, and prefer to have trust in the maintainers, release managers and developers to think about every patch and its consequences for the user-base and plan the releases accordingly (either separating the maintenance from the security issues or having them in one bunch).

4

Thank you for your insights Ernesto. I just wanted to highlight that I didn’t bring this up because of the controversies (or rather " controversies ") around the latest security update and regressions.
That’s why I chose the earlier 10.4.18 release as an example. The situation I described occurs for us with every security release - not just once every now and then.

5

Maybe it’s worth shifting the focus a bit… away from having full-blown releases just containing those security fixes - to instead finding a simpler way to provide patches (patch files and composer-patches per typo3/cms-* package).

This would address the demand, (hopefully) without adding too much overhead with additional version…

6

I was thinking you mean the situation where a maintenance release contains a regression or some “action” is required on your side instead of just installing the patch. That does not happen very often, which I what I meant with “trusting” the maintainers. You can usually install a TYPO3 patchlevel release and deploy it without even testing it - which is sufficient for 98% of the sites I could think of (being able to quickly rollback a deployment would be important).

For the remaining 2% where “price is not an issue” (where you have the time and money to test and analyze ever single fix for a patchlevel release manually), I can understand the concern.

To make this requirement a reality, there would in the end mean a branch with only security patches, since you never know how far away you are from the “latest stable patchlevel release” (considering there are regularly one patchlevel release per month, there is not much time in between to “squeeze” separate security fixes).

That could be a good compromise. We already have the patches available, i.e. https://github.com/typo3/typo3/commit/5a9f592733.patch - they are even linked in the release notes (https://get.typo3.org/release-notes/10.4.19).

So one could be fulfilling the requirement of “installing security fixes without updating the the latest patchlevel” without changing the current strategy.

We use https://github.com/cweagans/composer-patches to include patches at composer install time, which is able to fetch the patches via curl, i.e.:

  "extra": {
    "patches": {
      "typo3/cms": {
        "Security fix 10.4.19": "https://github.com/typo3/typo3/commit/5a9f592733"
      }
    }
  }

Drawback is of course that these patches could potentially contain merge conflicts when applied to older patchlevel releases, but that must be solved then by the ones requiring that instead of by a small team which has limited resources?

7

Are you applying every patch level release consecutively (10.4.15, 10.4.16, 10.4.17, 10.4.18) or “jumping” from one security release to the next security release, just focusing on the “more important” versions?

8

We apply all updates consecutively. The way our testing and release cycle works, the actual rollout to production is 2-4 weeks after the TYPO3 release. For TYPO3 security updates we always perform a risk assessment and decide whether it will also take the normal cycle or is shipped earlier.

9

Maybe a way to go would be to provide a single patch file which incorporates several security commits and targets certain branch e.g. v10 (so it applies without conflicts)?
Magento does sth like that.
This however also implies that this patched version has to be tested additionally to the “current branch” release.

From my POV the biggest bottleneck is that we’re short on testing power, especially using bigger real-life projects (which have end-to-end test suites) - to be able to catch all possible side effects.

I think this can be improved in two ways (and both should be taken)

  1. by having a few early adopter TYPO3 partners who (under NDA) could have access to the security patches before they are released and would run automatic test suite as well as manual tests.
    I see this as a win-win, the partner has to test security release anyway, so it’s not putting much more work for him. And gives the possibility to prevent regressions. And for the Core, it gives testing in real life scenarios.

  2. by having more end-to-end testes for the TYPO3 especially frontend rendering (recently there were first patches in that area)

10

Exactly. I could create such tool in case I receive reasonable funding for it. The idea is to provide a gerrit change set URL and the tool extracts individual patches for the composer patches plugin

11

I never install a security release without waiting a few days. Too often there are regressions in them.

I cannot says if these regression occurs because of the maintenance fixes or because the automated test cover not enough code.

For our companies workflow patch files will work fine. Though other users may prefer releases.

I always thought that security fixes come with maintenance changes to obscure the fixed vulnerabilities.

12

Here’s a monorepo-to-package splitter for composer-patches working with Gerrit Patch IDs: https://gist.github.com/xperseguers/ce58e2d27f1115278bf06e5a8eb4b983

13

As another data point, what Drupal does (did?) is what a few people have suggested above: Release a .z tag that includes the security fix plus whatever other bugs have been fixed in the meantime, and release raw patches for the most recent version for just the security fix. Those that want the security fix right-now without checking the bugfixes can just apply the patches, and then update to the bugfix at their leisure. Of course, patches are only provided for the most recent patch version, though they will often work on several versions back by chance.

Exceptions are made for really really big security fixes, where a point release dedicated to just that fix and postponing bugfixes to later, just to make applying the bugfix as drop-dead easy as possible. (I think that’s only been done once or twice, for the Drupageddon issues.)

That strategy would probably work here as well.

14

Publishing patch files for security fixes against the latest maintenance release, will not work in all situations, but I figure it would be a good compromise. We’d likely review and rollout those much quicker than a new release.