| 1 |
seanius |
187 |
[[!meta title="DEP-10: parallelized release management"]] |
| 2 |
seanius |
175 |
|
| 3 |
seanius |
187 |
Title: parallelized release management" |
| 4 |
seanius |
175 |
DEP: 10 |
| 5 |
|
|
State: DRAFT |
| 6 |
|
|
Date: 2011-04-30 |
| 7 |
|
|
Drivers: Sean Finney <seanius@debian.org>, |
| 8 |
|
|
Raphaƫl Hertzog <hertzog@debian.org> |
| 9 |
|
|
URL: http://dep.debian.net/deps/dep10 |
| 10 |
plessy |
284 |
Source: http://anonscm.debian.org/viewvc/dep/web/deps/dep10.mdwn |
| 11 |
seanius |
175 |
License: GPL |
| 12 |
seanius |
187 |
Abstract: Proposal for changes to release management methodology and |
| 13 |
|
|
infrastructure, allowing the Debian release process to function |
| 14 |
|
|
in parallel to non-release related updates. |
| 15 |
seanius |
175 |
|
| 16 |
seanius |
182 |
[[!toc levels=2]] |
| 17 |
seanius |
181 |
|
| 18 |
seanius |
176 |
# Introduction / Problem scope |
| 19 |
|
|
|
| 20 |
|
|
Currently, as the project nears a new stable release, a freeze is instituted |
| 21 |
|
|
on the testing suite. The freeze is put in place to allow the release team |
| 22 |
|
|
to focus on resolving the remaining Release Critical (RC) bugs for the next |
| 23 |
|
|
stable release, and at the same time to prevent regressions from new uploads. |
| 24 |
|
|
Typically the freeze begins as an advisory "soft freeze", which over |
| 25 |
|
|
time increases in strictness and levels of enforcement. |
| 26 |
|
|
|
| 27 |
|
|
Unsuprisingly, as the strictness of the freeze increases, there is an |
| 28 |
|
|
inversely proportional decrease in other non-release targeted maintainer |
| 29 |
|
|
activity. Since unstable still is the preferred route for packages to |
| 30 |
|
|
reach the new release during this period, maintainers are highly discouraged |
| 31 |
|
|
and in some cases prevented from doing non-release targeted activities |
| 32 |
|
|
in unstable. |
| 33 |
|
|
|
| 34 |
seanius |
190 |
<a name="introduction-probs"> |
| 35 |
seanius |
176 |
The reduction of such non-release activity is viewed as problematic in |
| 36 |
|
|
this DEP, for some inter-related reasons: |
| 37 |
|
|
|
| 38 |
|
|
* New features and innovations are put on hold, or at least not commonly |
| 39 |
|
|
available, until after the release is made. |
| 40 |
|
|
* Overall Maintainer activity decreases as freezes persist. |
| 41 |
|
|
* Potential userbase is lost (missing feature X, switch distro). |
| 42 |
|
|
* The volatility of testing/unstable increases (and thus quality decreases) |
| 43 |
|
|
with a deluge of new uploads after the freeze is lifted. |
| 44 |
|
|
|
| 45 |
|
|
As Debian is well known for taking a "release when it's ready" approach, the |
| 46 |
|
|
freeze periods are generally known to last considerable amounts of time. |
| 47 |
|
|
Consider the last three freezes: |
| 48 |
|
|
|
| 49 |
seanius |
183 |
* etch: 4 months |
| 50 |
seanius |
176 |
* lenny: 7 months |
| 51 |
|
|
* squeeze: 6 months |
| 52 |
|
|
|
| 53 |
|
|
This means, in rough terms, that the when testing thaws, that the Debian |
| 54 |
|
|
project may be starting from a state half a year behind comparable |
| 55 |
seanius |
189 |
distributions; the project is, in essence, starting from a 6 month |
| 56 |
|
|
standstill. |
| 57 |
seanius |
176 |
|
| 58 |
seanius |
189 |
This standstill, as well as the subsequent rush of "post-thaw" uploads, |
| 59 |
|
|
will introduce further delays to the next release, as release goals |
| 60 |
|
|
will be set relative to this handicapped starting point and a non-trivial |
| 61 |
|
|
amount of developer effort will be spent reconciling problems in the ensuing |
| 62 |
|
|
rush of uploads and transitions. |
| 63 |
seanius |
176 |
|
| 64 |
seanius |
178 |
# Past and present release methods |
| 65 |
|
|
|
| 66 |
|
|
## Frozen (< 2000) |
| 67 |
|
|
|
| 68 |
|
|
Before the introduction of testing, Debian used a simple release process |
| 69 |
|
|
where the unstable suite was snapshotted into a `frozen` suite. This suite |
| 70 |
|
|
would then be used exclusively for preparing the next stable release, with |
| 71 |
|
|
unstable continuing in parallel. |
| 72 |
|
|
|
| 73 |
|
|
Before freeze Freeze Release |
| 74 |
|
|
|
| 75 |
|
|
[unstable/sid]-------------------------------------------------------------- |
| 76 |
|
|
\ |
| 77 |
|
|
[frozen].-.-.-.-.[stable/R_N].-.-.-.-.-.-.-.-.-.-.-.-.-. |
| 78 |
|
|
|
| 79 |
|
|
[stable/R_N-1].-.-.-.-.-.-.-.-.-.-.-.[oldstable/R_N-1].-.-.-(EOL) |
| 80 |
|
|
|
| 81 |
seanius |
184 |
--------: Normal activity. Standard rules for uploads and migrations. |
| 82 |
|
|
.-.-.-.-: Release targeted activity. Freezes and limited uploads. |
| 83 |
|
|
\ \ \ \ : Package migration activity. |
| 84 |
seanius |
178 |
|
| 85 |
seanius |
179 |
### Use cases with `frozen` |
| 86 |
seanius |
178 |
|
| 87 |
seanius |
179 |
Users were divided into two sets, those using `unstable` and those using |
| 88 |
|
|
`stable`. Very few users would use both, as the two lines of development |
| 89 |
|
|
would quickly diverge from each other. |
| 90 |
seanius |
178 |
|
| 91 |
seanius |
179 |
### Benefits with `frozen` |
| 92 |
|
|
|
| 93 |
seanius |
178 |
* Work in unstable continued, unaffected by the freeze. |
| 94 |
|
|
|
| 95 |
seanius |
179 |
### Problems with `frozen` |
| 96 |
seanius |
178 |
|
| 97 |
|
|
* Release work started from a buggy suite. |
| 98 |
|
|
* Duplicated effort in unstable/frozen. |
| 99 |
|
|
* RM had more work/responsibilities, and was prone to burn-out. |
| 100 |
|
|
|
| 101 |
|
|
## Testing (2000-Present) |
| 102 |
|
|
|
| 103 |
|
|
The testing suite was introduced in Debian between the release of potato |
| 104 |
seanius |
183 |
and woody, in the fall of 2000[1][1]. The goal was to provide |
| 105 |
seanius |
178 |
a suite that was in a better state for release preparation, by having |
| 106 |
|
|
both automated and manual tools to keep down the level of bugs and |
| 107 |
|
|
general volatility. |
| 108 |
|
|
|
| 109 |
|
|
As a pleasant and convenient side-effect, the new suite also provided |
| 110 |
|
|
a "slightly less buggy unstable" for developers and end-users, who wanted |
| 111 |
|
|
newer software/features not available in stable, but wanted some level |
| 112 |
|
|
of protection to the relatively unpredictable nature of unstable. |
| 113 |
|
|
|
| 114 |
seanius |
184 |
Before release Freeze Release |
| 115 |
|
|
|
| 116 |
seanius |
185 |
[unstable/sid]----------.--.--.--.-.-.-.-.---------------------------------- |
| 117 |
seanius |
184 |
\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ |
| 118 |
seanius |
185 |
[testing/R_N]----------.-.-.-.-.-.-.-.-.-.-[testing/R_N+1]------------------ |
| 119 |
seanius |
184 |
/ / \ |
| 120 |
|
|
/ / [stable/R_N].-.-.-.-.-.-.-.-.-.-. |
| 121 |
|
|
/ / / / / |
| 122 |
seanius |
178 |
[R_N p-u].-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.- |
| 123 |
seanius |
184 |
|
| 124 |
seanius |
178 |
[stable/R_N-1].-.-.-.-.-.-.-.-.-.-.-.-.-.-[oldstable/R_N-1].-.-.-(EOL) |
| 125 |
seanius |
184 |
|
| 126 |
|
|
--------: Normal activity. Standard rules for uploads and migrations. |
| 127 |
|
|
.-.-.-.-: Release targeted activity. Freezes and limited uploads. |
| 128 |
|
|
\ \ \ \ : Package migration activity. Spacing of marks is a rough |
| 129 |
|
|
indication of frequency. |
| 130 |
seanius |
178 |
|
| 131 |
|
|
During the freeze, the testing suite becomes entirely dedicated to the |
| 132 |
|
|
release work. In practice, this also means that unstable is also to |
| 133 |
|
|
some extent frozen from non-release activity for many packages, since |
| 134 |
|
|
it still serves as the main route to testing for new uploads. |
| 135 |
|
|
|
| 136 |
seanius |
179 |
### Use cases with `testing` |
| 137 |
seanius |
178 |
|
| 138 |
seanius |
179 |
The introduction of the new suite also introduced a type of continuum |
| 139 |
|
|
in which users now have more flexibility in selecting what to have installed. |
| 140 |
|
|
|
| 141 |
|
|
* `unstable`: only the "bleeding edge" of Debian updates. |
| 142 |
|
|
* `testing`: only the "leading edge" of Debian updates. |
| 143 |
|
|
* `testing/unstable`: A hybrid solution of the two previous use cases, |
| 144 |
|
|
allowing for a reasonable balance between them. |
| 145 |
seanius |
188 |
* `stabe`: The latest stable release. |
| 146 |
seanius |
179 |
* `stable/testing`: the stable release with perhaps some newer packages |
| 147 |
|
|
installed, typically also making use of "APT pins" to prevent unwanted |
| 148 |
seanius |
188 |
upgraes. Less common now with the inclusion of backports and volatile |
| 149 |
seanius |
179 |
as official Debian services. |
| 150 |
|
|
|
| 151 |
seanius |
180 |
### Benefits with `testing` |
| 152 |
seanius |
179 |
|
| 153 |
seanius |
178 |
* Release branch has far fewer bugs at the start of the release process. |
| 154 |
|
|
* (Ideally) shorter release process with fewer problems. |
| 155 |
|
|
* Large user-base for testing stable-targeted fixes. |
| 156 |
|
|
|
| 157 |
seanius |
179 |
### Problems with `testing` |
| 158 |
seanius |
178 |
|
| 159 |
seanius |
190 |
(See [Introduction](#introduction-probs)) |
| 160 |
seanius |
178 |
|
| 161 |
seanius |
188 |
# Constraints and requirements for any change to the release process |
| 162 |
|
|
|
| 163 |
|
|
Based on feedback on the `debian-devel` mailing list, this proposal |
| 164 |
|
|
makes a number of assumptions about constraints and requirements for |
| 165 |
|
|
any alteration of the current release process. |
| 166 |
|
|
|
| 167 |
seanius |
190 |
1. Any irreconcilable conflict should side towards release preparation |
| 168 |
|
|
* activity which can not be done in parellel must only be done for |
| 169 |
|
|
release preparation. |
| 170 |
|
|
* any proposal must be no less safe than the current system with |
| 171 |
|
|
regards to human error (uploads to wrong suite, etc, un-unannounced |
| 172 |
|
|
transitions, etc). |
| 173 |
|
|
1. Debian stable releases need sufficient test coverage by users |
| 174 |
|
|
* a testing (or testing-like) quarantine is needed for QA purposes |
| 175 |
|
|
* sufficient users should be using this quarantine to give good coverage. |
| 176 |
|
|
1. Any parallel work should interfere minimally with release preparations |
| 177 |
|
|
* newer software should not prevent an upload path to the release. |
| 178 |
|
|
* library transitions and similar should not cause unreasonable overhead. |
| 179 |
|
|
* it must still be possible to remove packages from the release. |
| 180 |
|
|
1. The upgrade path and archive state must be consistant and reconcilable |
| 181 |
|
|
* packages updated outside of the release should always have higher versions |
| 182 |
|
|
than in the release, even if both were updated. |
| 183 |
|
|
* If the suite contents are to change post-release, the proposal must |
| 184 |
|
|
account for renamed/duplicate/removed packagets. |
| 185 |
seanius |
188 |
|
| 186 |
seanius |
190 |
# A host of proposals |
| 187 |
seanius |
188 |
|
| 188 |
seanius |
190 |
During the post-squeeze release feedback on debian-devel, there has been a |
| 189 |
|
|
number of ideas and suggestions for how the freeze of the release process |
| 190 |
|
|
could be minimized, circumvented, or avoided altogether. Some of the more |
| 191 |
|
|
popular and/or controversial alternatives will be discussed and analyzed |
| 192 |
|
|
in the following section. |
| 193 |
seanius |
188 |
|
| 194 |
seanius |
190 |
## Branch `testing` at freeze time (a.k.a `frozen v2.0`) |
| 195 |
seanius |
188 |
|
| 196 |
seanius |
190 |
At some point during release preparation, somewhere between the "soft" and |
| 197 |
|
|
"hard" freeze points of the current release, the `testing` suite is split |
| 198 |
|
|
into two new independant suites. The first of these suites is used for |
| 199 |
|
|
continuing the (frozen) next stable release, while the other continues |
| 200 |
|
|
receiving updates via `unstable`. |
| 201 |
seanius |
188 |
|
| 202 |
seanius |
190 |
Before release Freeze Release |
| 203 |
seanius |
188 |
|
| 204 |
seanius |
190 |
[unstable/sid]-------------------------------------------------------------- |
| 205 |
|
|
\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ |
| 206 |
|
|
[testing/R_N]----------[testing/R_N+1]-------------------------------------- |
| 207 |
|
|
\ \ \ \ \ \ |
| 208 |
|
|
[R_N].-.-.-.-.-.-.[stable/R_N].-.-.-.-.-.-.-.-.- |
| 209 |
|
|
/ / / / / / / / / |
| 210 |
|
|
[R_N p-u].-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.- |
| 211 |
|
|
|
| 212 |
|
|
[stable/R_N-1].-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-[oldstable/R_N-1]-.-(EOL) |
| 213 |
seanius |
188 |
|
| 214 |
seanius |
190 |
--------: Normal activity. Standard rules for uploads and migrations. |
| 215 |
|
|
.-.-.-.-: Release targeted activity. Freezes and limited uploads. |
| 216 |
|
|
\ \ \ \ : Package migration activity. Spacing of marks is a rough |
| 217 |
|
|
indication of frequency. |
| 218 |
seanius |
188 |
|
| 219 |
|
|
|
| 220 |
seanius |
190 |
This is very similar to the previous `frozen` release process, though |
| 221 |
|
|
with the key difference that at the point the branch-off occurs, the |
| 222 |
|
|
suite is in a far better condition due to the additional and continuous |
| 223 |
|
|
QA recieved in the `unstable`->`testing` process. |
| 224 |
|
|
|
| 225 |
|
|
As a consequence, work in `unstable` will quickly diverge from the state |
| 226 |
|
|
of the frozen release, which has to immediate consequences: |
| 227 |
|
|
* The release team will rely on using the corresponding `-proposed-updates` |
| 228 |
|
|
pseudo-suite for release-targeted uploads, as undesired transitions/updates |
| 229 |
|
|
in `unstable` will quickly pare off the candidates for direct migration. |
| 230 |
|
|
* The default path (and userbase) of `unstable`->`testing` can no longer |
| 231 |
|
|
be relied upon for the same amount of QA in the release preparation |
| 232 |
|
|
process. |
| 233 |
|
|
|
| 234 |
|
|
### Benefits |
| 235 |
|
|
### Problems |
| 236 |
|
|
### Conclusion |
| 237 |
|
|
|
| 238 |
|
|
## Implement an `unstable-updates` |
| 239 |
|
|
|
| 240 |
|
|
A new pseudo-suite `unstable-updates` (or some other creative name left |
| 241 |
|
|
for a later exercise) is established, which functions in a manner similar |
| 242 |
|
|
to the existing `experimental` pseudo-suite. A key difference from |
| 243 |
|
|
`experimental` is that this suite is explicitly expected to feed packages |
| 244 |
|
|
to the `unstable` suite, and thus the contents should be subject to the |
| 245 |
|
|
same quality standards expected of `unstable` uploads. Outside of the |
| 246 |
|
|
release process, this feeding can be accomplished either by automatic |
| 247 |
|
|
`britney` migrations or some form of semi-automatic "gating" process |
| 248 |
|
|
requiring ACK's from either the maintainer or the release team. |
| 249 |
|
|
|
| 250 |
|
|
During freeze, which continues as it does today, the release team disables |
| 251 |
|
|
the migrations from `unstable-updates` to `unstable`. The former, in |
| 252 |
|
|
effect, becomes a staging area for updates to the *following* release. |
| 253 |
|
|
Post-release, the migration is re-activated, possibly with some additional |
| 254 |
|
|
controls allowing the updates to occur in a more orderly fashion if needed. |
| 255 |
|
|
|
| 256 |
|
|
Before release Freeze Release |
| 257 |
|
|
|
| 258 |
|
|
[unstable-updates]---------------------------------------------------------- |
| 259 |
|
|
\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ |
| 260 |
|
|
[unstable/sid]----------.--.--.--.-.-.-.-.---------------------------------- |
| 261 |
|
|
\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ |
| 262 |
|
|
[testing/R_N]----------.-.-.-.-.-.-.-.-.-.-[testing/R_N+1]------------------ |
| 263 |
|
|
/ / \ |
| 264 |
|
|
/ / [stable/R_N].-.-.-.-.-.-.-.-.-.-. |
| 265 |
|
|
/ / / / / |
| 266 |
|
|
[R_N p-u].-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.- |
| 267 |
|
|
|
| 268 |
|
|
[stable/R_N-1].-.-.-.-.-.-.-.-.-.-.-.-.-.-[oldstable/R_N-1].-.-.-(EOL) |
| 269 |
|
|
|
| 270 |
|
|
--------: Normal activity. Standard rules for uploads and migrations. |
| 271 |
|
|
.-.-.-.-: Release targeted activity. Freezes and limited uploads. |
| 272 |
|
|
\ \ \ \ : Package migration activity. Spacing of marks is a rough |
| 273 |
|
|
indication of frequency. |
| 274 |
|
|
|
| 275 |
|
|
|
| 276 |
|
|
### Benefits |
| 277 |
|
|
### Problems |
| 278 |
|
|
### Conclusion |
| 279 |
|
|
|
| 280 |
|
|
## Implement "Debian PPA's" and advocate their use |
| 281 |
|
|
|
| 282 |
|
|
Another alternative which has been brought up at several points in |
| 283 |
|
|
discussion is the use of *Personal Package Archives*, a.k.a. *PPA*'s, |
| 284 |
|
|
a.k.a. *DSR*'s, as a means to provide packages updates entirely outside |
| 285 |
|
|
of the release process. Individual maintainers, or teams of related |
| 286 |
|
|
projects, could establish and host independant and self-contained |
| 287 |
|
|
repositories for newer packages. |
| 288 |
|
|
|
| 289 |
|
|
During a release freeze, these repositories are created on an ad-hoc |
| 290 |
|
|
basis to host any updated packages not targeted for release. Users seeking |
| 291 |
|
|
newer software would be directed to add additional APT sources to their |
| 292 |
|
|
package manager configuration to enable the updates. |
| 293 |
|
|
|
| 294 |
|
|
Additionally, PPA's could serve several other purposes apart from avoiding |
| 295 |
|
|
the testing freeze. For example, Outside of the release proucess, they |
| 296 |
|
|
can continue to host newer packages than are immediately available in |
| 297 |
|
|
unstable, in a manner similar to the existing `experimental` pseudo-suite. |
| 298 |
|
|
|
| 299 |
|
|
In a manner similar to the `backports.org` services, PPA's could be |
| 300 |
|
|
established to serve self-contained sets of backported packages to |
| 301 |
|
|
previous stable releases. For example, newer versions of popular |
| 302 |
|
|
desktop software, or entirely desktop environments, could be hosted |
| 303 |
|
|
in these self-contained release areas. |
| 304 |
|
|
|
| 305 |
|
|
PPA's could also be used by maintainers and/or the release team for |
| 306 |
|
|
preparation of package transitions outside of `unstable`, providing an |
| 307 |
|
|
orderly way to update sets of interdependant packages and reducing |
| 308 |
|
|
overall breakage. This would lead to not only a higher quality experience |
| 309 |
|
|
in `testing/unstable`, but also streamline the process for subsequent |
| 310 |
|
|
releases, which would experience fewer blockages in work due to |
| 311 |
|
|
ongoing transitions. |
| 312 |
|
|
|
| 313 |
|
|
### Benefits |
| 314 |
|
|
### Problems |
| 315 |
|
|
### Conclusion |
| 316 |
|
|
|
| 317 |
|
|
## Implement a `rolling` release |
| 318 |
|
|
|
| 319 |
|
|
**NOTE**: this section needs to be re-worked. We're not talking about |
| 320 |
|
|
implementing rolling, though there are some key points which do need to |
| 321 |
|
|
be discussed in how this implementation would affect and work with rolling. |
| 322 |
|
|
The idea of a Debian `rolling` release has been discussed with increased |
| 323 |
|
|
amounts of interest and seriousness during and after the `squeeze` release |
| 324 |
|
|
cycle. |
| 325 |
|
|
|
| 326 |
|
|
While it has been proposed in several shapes and forms, the overarching |
| 327 |
|
|
concept is that users should be provided a constantly updating release |
| 328 |
|
|
similar to the current `testing` suite, with some key differences: |
| 329 |
|
|
|
| 330 |
|
|
* ability to avoid/circumvent blockage from ongoing transitions in |
| 331 |
|
|
`unstable`. |
| 332 |
|
|
* a more official level of support (security, RC bug fixes, etc) |
| 333 |
|
|
* continuous updates, even during release freezes |
| 334 |
|
|
|
| 335 |
|
|
It's worth note that the motivation/proposal for `rolling` doesn't |
| 336 |
|
|
entirely overlap with the motivations behind the DEP. Certainly both |
| 337 |
|
|
proposals involve the ability to have continuous updates during the |
| 338 |
|
|
entire release cycle, but beyond that the additional desires regarding |
| 339 |
|
|
rolling are orthogonal (or only related in as much as they might place |
| 340 |
|
|
constraints on the complementing implementations) |
| 341 |
|
|
|
| 342 |
|
|
Furthermore, as stated, there have been several *different* |
| 343 |
|
|
visions/implementations of `rolling` discussed. |
| 344 |
|
|
|
| 345 |
|
|
### `rolling` by way of "alternate entry points" |
| 346 |
|
|
|
| 347 |
|
|
A new and entirely independant `rolling` suite is established, in parallel |
| 348 |
|
|
to testing. Both suites have a default entry point of the `unstable` suite, |
| 349 |
|
|
which feeds both suites through a similar manner. |
| 350 |
|
|
|
| 351 |
|
|
While packages may migrate into rolling via typical `unstable` |
| 352 |
|
|
uploads, there also exist means to directly upload to the suite via a |
| 353 |
|
|
`rolling-proposed-updates` or similarly named pseudo-suite. This |
| 354 |
|
|
pseudo suite would be an overlay on top of the standard `rolling` suite, |
| 355 |
|
|
allowing for the testing and acceptance of direct fixes without the |
| 356 |
|
|
risk of being blocked by any ongoing transitions or otherwise unsatisfiable |
| 357 |
|
|
dependency chains. |
| 358 |
|
|
|
| 359 |
|
|
During a release freeze, |
| 360 |
|
|
as the default entry point for `rolling`, |
| 361 |
|
|
|
| 362 |
|
|
`rolling-proposed-updates` could take the |
| 363 |
|
|
additional role of accepting new non-release targeted uploads, or |
| 364 |
|
|
an additional entry point could be defined to replace the role of |
| 365 |
|
|
unstable. |
| 366 |
|
|
|
| 367 |
|
|
### `rolling` by way of "override hints" |
| 368 |
|
|
|
| 369 |
|
|
|
| 370 |
|
|
|
| 371 |
|
|
### Benefits |
| 372 |
|
|
### Problems |
| 373 |
|
|
### Conclusion |
| 374 |
|
|
|
| 375 |
|
|
|
| 376 |
|
|
|
| 377 |
seanius |
178 |
[1]: http://lists.debian.org/debian-devel/2000/08/msg00906.html |