[[!meta title="DEP-10: parallelized release management"]]

    Title: parallelized release management"
    DEP: 10
    State: DRAFT
    Date: 2011-04-30
    Drivers: Sean Finney <seanius@debian.org>,
	 Raphaël Hertzog <hertzog@debian.org>
    URL: http://dep.debian.net/deps/dep10
    Source: http://anonscm.debian.org/viewvc/dep/web/deps/dep10.mdwn
    License: GPL
    Abstract: Proposal for changes to release management methodology and 
     infrastructure, allowing the Debian release process to function
     in parallel to non-release related updates.

[[!toc levels=2]]

# Introduction / Problem scope

Currently, as the project nears a new stable release, a freeze is instituted
on the testing suite.  The freeze is put in place to allow the release team
to focus on resolving the remaining Release Critical (RC) bugs for the next
stable release, and at the same time to prevent regressions from new uploads.
Typically the freeze begins as an advisory "soft freeze", which over
time increases in strictness and levels of enforcement.

Unsuprisingly, as the strictness of the freeze increases, there is an
inversely proportional decrease in other non-release targeted maintainer
activity.  Since unstable still is the preferred route for packages to
reach the new release during this period, maintainers are highly discouraged
and in some cases prevented from doing non-release targeted activities
in unstable.

<a name="introduction-probs">
The reduction of such non-release activity is viewed as problematic in
this DEP, for some inter-related reasons:

 * New features and innovations are put on hold, or at least not commonly
   available, until after the release is made.
 * Overall Maintainer activity decreases as freezes persist.
 * Potential userbase is lost (missing feature X, switch distro).
 * The volatility of testing/unstable increases (and thus quality decreases)
   with a deluge of new uploads after the freeze is lifted.

As Debian is well known for taking a "release when it's ready" approach, the
freeze periods are generally known to last considerable amounts of time.
Consider the last three freezes:

 * etch: 4 months
 * lenny: 7 months
 * squeeze: 6 months

This means, in rough terms, that the when testing thaws, that the Debian
project may be starting from a state half a year behind comparable
distributions; the project is, in essence, starting from a 6 month
standstill.  

This standstill, as well as the subsequent rush of "post-thaw" uploads,
will introduce further delays to the next release, as release goals
will be set relative to this handicapped starting point and a non-trivial
amount of developer effort will be spent reconciling problems in the ensuing
rush of uploads and transitions.

# Past and present release methods

## Frozen (< 2000)

Before the introduction of testing, Debian used a simple release process
where the unstable suite was snapshotted into a `frozen` suite.  This suite
would then be used exclusively for preparing the next stable release, with
unstable continuing in parallel.

    Before freeze       Freeze           Release

    [unstable/sid]--------------------------------------------------------------
                        \
                        [frozen].-.-.-.-.[stable/R_N].-.-.-.-.-.-.-.-.-.-.-.-.-.
     
    [stable/R_N-1].-.-.-.-.-.-.-.-.-.-.-.[oldstable/R_N-1].-.-.-(EOL)

    --------: Normal activity.  Standard rules for uploads and migrations.
    .-.-.-.-: Release targeted activity.  Freezes and limited uploads. 
    \ \ \ \ : Package migration activity.

### Use cases with `frozen`

Users were divided into two sets, those using `unstable` and those using
`stable`.  Very few users would use both, as the two lines of development
would quickly diverge from each other.

### Benefits with `frozen`

 * Work in unstable continued, unaffected by the freeze.

### Problems with `frozen`

 * Release work started from a buggy suite.
 * Duplicated effort in unstable/frozen.
 * RM had more work/responsibilities, and was prone to burn-out.

## Testing (2000-Present)

The testing suite was introduced in Debian between the release of potato
and woody, in the fall of 2000[1][1].  The goal was to provide
a suite that was in a better state for release preparation, by having
both automated and manual tools to keep down the level of bugs and
general volatility.

As a pleasant and convenient side-effect, the new suite also provided
a "slightly less buggy unstable" for developers and end-users, who wanted
newer software/features not available in stable, but wanted some level
of protection to the relatively unpredictable nature of unstable.

    Before release         Freeze              Release
    
    [unstable/sid]----------.--.--.--.-.-.-.-.----------------------------------
         \ \ \ \ \ \ \ \ \  \   \    \       \   \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \
    [testing/R_N]----------.-.-.-.-.-.-.-.-.-.-[testing/R_N+1]------------------
                                          / / \
                                         / /   [stable/R_N].-.-.-.-.-.-.-.-.-.-.
                                        / /          /          /           /
                                   [R_N p-u].-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-
    
    [stable/R_N-1].-.-.-.-.-.-.-.-.-.-.-.-.-.-[oldstable/R_N-1].-.-.-(EOL)
    
    --------: Normal activity.  Standard rules for uploads and migrations.
    .-.-.-.-: Release targeted activity.  Freezes and limited uploads. 
    \ \ \ \ : Package migration activity.  Spacing of marks is a rough
              indication of frequency.

During the freeze, the testing suite becomes entirely dedicated to the
release work.  In practice, this also means that unstable is also to
some extent frozen from non-release activity for many packages, since
it still serves as the main route to testing for new uploads.

### Use cases with `testing`

The introduction of the new suite also introduced a type of continuum
in which users now have more flexibility in selecting what to have installed.

 * `unstable`: only the "bleeding edge" of Debian updates.
 * `testing`: only the "leading edge" of Debian updates.
 * `testing/unstable`: A hybrid solution of the two previous use cases,
    allowing for a reasonable balance between them.
 * `stabe`: The latest stable release.
 * `stable/testing`: the stable release with perhaps some newer packages
    installed, typically also making use of "APT pins" to prevent unwanted
    upgraes.  Less common now with the inclusion of backports and volatile
	as official Debian services.

### Benefits with `testing`

 * Release branch has far fewer bugs at the start of the release process.
 * (Ideally) shorter release process with fewer problems.
 * Large user-base for testing stable-targeted fixes.

### Problems with `testing`

(See [Introduction](#introduction-probs))

# Constraints and requirements for any change to the release process

Based on feedback on the `debian-devel` mailing list, this proposal
makes a number of assumptions about constraints and requirements for
any alteration of the current release process.

 1. Any irreconcilable conflict should side towards release preparation
  * activity which can not be done in parellel must only be done for
    release preparation.
  * any proposal must be no less safe than the current system with
    regards to human error (uploads to wrong suite, etc, un-unannounced
    transitions, etc).
 1. Debian stable releases need sufficient test coverage by users
  * a testing (or testing-like) quarantine is needed for QA purposes
  * sufficient users should be using this quarantine to give good coverage. 
 1. Any parallel work should interfere minimally with release preparations
  * newer software should not prevent an upload path to the release.
  * library transitions and similar should not cause unreasonable overhead.
  * it must still be possible to remove packages from the release.
 1. The upgrade path and archive state must be consistant and reconcilable
  * packages updated outside of the release should always have higher versions
    than in the release, even if both were updated.
  * If the suite contents are to change post-release, the proposal must
    account for renamed/duplicate/removed packagets.

# A host of proposals

During the post-squeeze release feedback on debian-devel, there has been a
number of ideas and suggestions for how the freeze of the release process
could be minimized, circumvented, or avoided altogether.  Some of the more
popular and/or controversial alternatives will be discussed and analyzed
in the following section.

## Branch `testing` at freeze time (a.k.a `frozen v2.0`)

At some point during release preparation, somewhere between the "soft" and
"hard" freeze points of the current release, the `testing` suite is split
into two new independant suites.  The first of these suites is used for
continuing the (frozen) next stable release, while the other continues
receiving updates via `unstable`.

    Before release         Freeze                 Release

    [unstable/sid]--------------------------------------------------------------
              \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \
    [testing/R_N]----------[testing/R_N+1]--------------------------------------
                               \ \ \  \   \    \
                                [R_N].-.-.-.-.-.-.[stable/R_N].-.-.-.-.-.-.-.-.-
                                 /    /   /  / / / /         /                /
                           [R_N p-u].-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-
     
    [stable/R_N-1].-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-[oldstable/R_N-1]-.-(EOL)

    --------: Normal activity.  Standard rules for uploads and migrations.
    .-.-.-.-: Release targeted activity.  Freezes and limited uploads. 
    \ \ \ \ : Package migration activity.  Spacing of marks is a rough
              indication of frequency.


This is very similar to the previous `frozen` release process, though
with the key difference that at the point the branch-off occurs, the
suite is in a far better condition due to the additional and continuous
QA recieved in the `unstable`->`testing` process.

As a consequence, work in `unstable` will quickly diverge from the state
of the frozen release, which has to immediate consequences:
 * The release team will rely on using the corresponding `-proposed-updates`
   pseudo-suite for release-targeted uploads, as undesired transitions/updates
   in `unstable` will quickly pare off the candidates for direct migration.
 * The default path (and userbase) of `unstable`->`testing` can no longer
   be relied upon for the same amount of QA in the release preparation
   process.

### Benefits
### Problems
### Conclusion

## Implement an `unstable-updates`

A new pseudo-suite `unstable-updates` (or some other creative name left
for a later exercise) is established, which functions in a manner similar
to the existing `experimental` pseudo-suite.  A key difference from
`experimental` is that this suite is explicitly expected to feed packages
to the `unstable` suite, and thus the contents should be subject to the
same quality standards expected of `unstable` uploads.  Outside of the
release process, this feeding can be accomplished either by automatic
`britney` migrations or some form of semi-automatic "gating" process
requiring ACK's from either the maintainer or the release team.

During freeze, which continues as it does today, the release team disables
the migrations from `unstable-updates` to `unstable`.  The former, in
effect, becomes a staging area for updates to the *following* release.
Post-release, the migration is re-activated, possibly with some additional
controls allowing the updates to occur in a more orderly fashion if needed.

    Before release         Freeze              Release
    
    [unstable-updates]----------------------------------------------------------
         \ \ \ \ \ \ \ \ \      \            \   \   \ \ \ \ \ \ \ \ \ \ \ \ \ \
    [unstable/sid]----------.--.--.--.-.-.-.-.----------------------------------
         \ \ \ \ \ \ \ \ \  \   \    \       \   \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \
    [testing/R_N]----------.-.-.-.-.-.-.-.-.-.-[testing/R_N+1]------------------
                                          / / \
                                         / /   [stable/R_N].-.-.-.-.-.-.-.-.-.-.
                                        / /          /          /           /
                                   [R_N p-u].-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-
    
    [stable/R_N-1].-.-.-.-.-.-.-.-.-.-.-.-.-.-[oldstable/R_N-1].-.-.-(EOL)
    
    --------: Normal activity.  Standard rules for uploads and migrations.
    .-.-.-.-: Release targeted activity.  Freezes and limited uploads. 
    \ \ \ \ : Package migration activity.  Spacing of marks is a rough
              indication of frequency.


### Benefits
### Problems
### Conclusion

## Implement "Debian PPA's" and advocate their use

Another alternative which has been brought up at several points in
discussion is the use of *Personal Package Archives*, a.k.a. *PPA*'s,
a.k.a. *DSR*'s, as a means to provide packages updates entirely outside
of the release process.  Individual maintainers, or teams of related
projects, could establish and host independant and self-contained
repositories for newer packages.

During a release freeze, these repositories are created on an ad-hoc
basis to host any updated packages not targeted for release.  Users seeking
newer software would be directed to add additional APT sources to their
package manager configuration to enable the updates.

Additionally, PPA's could serve several other purposes apart from avoiding
the testing freeze.  For example, Outside of the release proucess, they
can continue to host newer packages than are immediately available in
unstable, in a manner similar to the existing `experimental` pseudo-suite.

In a manner similar to the `backports.org` services, PPA's could be
established to serve self-contained sets of backported packages to
previous stable releases.  For example, newer versions of popular
desktop software, or entirely desktop environments, could be hosted
in these self-contained release areas.

PPA's could also be used by maintainers and/or the release team for
preparation of package transitions outside of `unstable`, providing an
orderly way to update sets of interdependant packages and reducing
overall breakage.  This would lead to not only a higher quality experience
in `testing/unstable`, but also streamline the process for subsequent
releases, which would experience fewer blockages in work due to
ongoing transitions.

### Benefits
### Problems
### Conclusion

## Implement a `rolling` release

**NOTE**: this section needs to be re-worked.  We're not talking about
implementing rolling, though there are some key points which do need to
be discussed in how this implementation would affect and work with rolling.
The idea of a Debian `rolling` release has been discussed with increased
amounts of interest and seriousness during and after the `squeeze` release
cycle.  

While it has been proposed in several shapes and forms, the overarching
concept is that users should be provided a constantly updating release
similar to the current `testing` suite, with some key differences:

 * ability to avoid/circumvent blockage from ongoing transitions in
  `unstable`.
 * a more official level of support (security, RC bug fixes, etc)
 * continuous updates, even during release freezes

It's worth note that the motivation/proposal for `rolling` doesn't
entirely overlap with the motivations behind the DEP.  Certainly both
proposals involve the ability to have continuous updates during the
entire release cycle, but beyond that the additional desires regarding
rolling are orthogonal (or only related in as much as they might place
constraints on the complementing implementations)

Furthermore, as stated, there have been several *different*
visions/implementations of `rolling` discussed.  

### `rolling` by way of "alternate entry points"

A new and entirely independant `rolling` suite is established, in parallel
to testing.  Both suites have a default entry point of the `unstable` suite,
which feeds both suites through a similar manner.

While packages may migrate into rolling via typical `unstable`
uploads, there also exist means to directly upload to the suite via a
`rolling-proposed-updates` or similarly named pseudo-suite.  This
pseudo suite would be an overlay on top of the standard `rolling` suite,
allowing for the testing and acceptance of direct fixes without the
risk of being blocked by any ongoing transitions or otherwise unsatisfiable
dependency chains.

During a release freeze, 
as the default entry point for `rolling`, 

`rolling-proposed-updates` could take the
additional role of accepting new non-release targeted uploads, or
an additional entry point could be defined to replace the role of
unstable.

### `rolling` by way of "override hints"



### Benefits
### Problems
### Conclusion



[1]: http://lists.debian.org/debian-devel/2000/08/msg00906.html