web/deps/dep2.mdwn

[[!meta title="DEP-2: Debian Package Maintenance Hub"]]

    Title: Debian Package Maintenance Hub
    DEP: 2
    State: DRAFT
    Date: 2012-01-13
    Drivers: Raphael Hertzog <hertzog@debian.org>
    URL: http://dep.debian.net/deps/dep2
    Source: http://anonscm.debian.org/viewvc/dep/web/deps/dep2.mdwn
    Abstract:
     Debian maintainers rely on a multitude of services (DDPO, PTS,
     DDPO-by-mail, BTS, etc), and information sources, in order to do
     their job. The flow of information varies greatly from case to case.
     .
     This proposal is about creating a central infrastructure
     that would consolidate several of those services and that would
     standardize the information flow.


Rationale
---------
This new package maintenance infrastructure is needed:

 * to fix long standing problems;
 * to provide a clean base to implement new features:
   * that will help maintainers do a better job;
   * that will help packaging teams to organize themselves;
   * that will help the QA team to ensure that all Debian packages are
     well maintained.

### Problems to solve

#### Maintainer vs Uploader

The flow of information is not the same depending on whether you're
listed in the Maintainer field (in which case most services mail you
directly) or not (in which case you're supposed to subscribe via the PTS
or via a dedicated mailing list). But the opposite is true as well, some
information is only available via the PTS and many maintainers have to
subscribe to the PTS while excluding almost everything just to get
the information they want.

See also [#507288](http://bugs.debian.org/507288) for some more
discussions on this topic.

This makes it very painful to change/switch the Maintainer field because
people have to update their PTS subscriptions accordingly.

#### Duplication of work / inconsistency between the DDPO and the PTS

The DDPO and the PTS are completely separate services. This leads to
duplication of work when a new information needs to be made available in
their respective interface. It can also lead to inconsistencies between
both services when bug occurs or when different choices are made.

#### Mailing lists as Maintainer

We often have mailing lists listed in the Maintainer field and it's not
clear who are the real package maintainers and how many of them there are.
The Uploaders field is often outdated, and/or is just a representation of
who worked last on the package instead of who feels responsible for the
package.

### Goals

#### Provide a working replacement for DDPO/PTS

Since the service aims to merge the DDPO and the PTS, it must be a working
replacement of both services and its set of features must englobe the
features of the actual services.

#### Replace maintenance mailing lists

Packaging teams often separate the mailing list that gets the bug traffic and
other notifications from their main discussion mailing list. This new
infrastructure should entirely replace the former kind of mailing lists.
Anybody receiving notifications and information directed to the package
maintainer should get them via this new infrastructure.

It allows us to know how many people are notified for a given problem.
If nobody is notified, the package is effectively orphaned. A more
interesting case to detect is when several persons are being notified but
all of them are MIA or marked as not being available for Debian (busy/in
vacation).

#### Keep track of the list of maintainers

Packages using this infrastructure should automatically keep track
of the list of their maintainers because anyone subscribing to a package
must pick a "role":

 * "maintainer" (only DD can select this role, or DM for their own packages)
 * "contributor"
 * "follower" (aka lurker)

#### Replace WNPP's RFH, RFA, O

Since the infrastructure already stores the information about who is
maintaining what, it only makes sense to extend it to provide the
list of orphaned packages (i.e. packages without maintainers).

RFH should be replaced by a system where the help request is better
formalized so that we can better direct new contributors in places
where their skills would be well used. Instead of just requesting
help, you could request:

 * a new maintainer (to replace you, i.e. RFA)
 * a supplementary co-maintainer
 * a bug triager
 * a C/Python/Perl/… programmer (you should be able to choose the programming
   language)

Each request also documents whether there's an associated offer of
"mentorship" associated to the help request. Of course, there would
also be a free form description to give more details about what's
expected.

#### Replace the LowThresholdNmu list

The [LowThresholdNmu](http://wiki.debian.org/LowThresholdNmu) wiki page
is a hack to let people know when NMU are welcome and not frowned
upon. This information should be properly stored in the database
and it should be associated to each package.

#### Provide better integration with the packaging VCS

A majority of packages are maintained in a VCS nowadays. It's thus
important to extract the relevant information and make it easily
accessible in the new interface.

Commits should appear in the usual activity stream of the package. We
should be able to instantly see whether the VCS contains pending changes
or not, or whether a new upstream version is in preparation there.

#### Enable new interactions with maintainers

The central role of this new "communication infrastructure" makes it
possible to design new interactions with maintainers. Instead of being
only a source of information, the infrastructure could be used to query
package maintainers and/or let them provide supplementary information.

This could be used to improve the MIA tracking process.

This infrastructure would also be a more natural place to store the
"available for Debian work" boolean flag ("vacation") that's currently
never used because it's buried in db.debian.org and that it's not
practical to update it.

This infrastructure could also be used to let maintainers document the
responsibilities that they have agreed to endorse, and describe the
associated commitments. That way it would be easier to detect packages
that cannot be well maintained because the set of maintainers do not cover
all the tasks that must be assumed to have a properly maintained package.

#### Provide new services to packaging teams

Given that this infrastructure would have native support for packaging
teams, it would also be a good place to offer some standardized services
for them.

For example, one of the central tool for teams are their VCS and it can be
useful for teams to be able to monitor the state of their package in the
VCS. The [PET](http://pet.alioth.debian.org/) tool could be adapted,
integrated and made available to all teams by default.

#### Support alternate notification systems

Email is the only official media used to communicate information to
Debian package maintainers. If all the relevant mails are going through
a central service, it's possible to store those emails and to forward
the relevant information by other means (RSS, XMPP, IRC, etc.). Also
new maintainers can then have access to some historic information
that used to be private for no good reasons.


High-level design of the new infrastructure
-------------------------------------------

### Fixing the flow of information

In order to cleanly solve the problem of the information flow, and to get
rid of the hacks made everywhere to send a copy of the mails to the PTS,
packages would be (progressively) modified to indicate
“Maintainer: &lt;source&gt;@pkgmaint.debian.org” in their control file.
The current content of the field would be moved to the “Uploaders” field.

Until all packages have been converted, the PTS would forward copies of
the mails to ensure that the new infrastucture can still be used for all
packages (even those who have not been updated yet).

Using this intermediary address also solves the problem of maintainers who
orphan their packages and are still listed as maintainers in many released
packages.

QUESTION: It would be cleaner if DAK, the BTS, and all relevant services,
could stop sending mails to the Maintainer field, and would instead always
send the mail to the new infrastructure. That way we wouldn't need to
change the Maintainer field and there would be no transition period.

The new infrastructure would then be configured with initial subscriptions
for emails listed in Maintainer fields (except for mailing lists, since
the infrastructure aims to replace them).

### Leveraging UDD

At least the PTS has been parsing Sources/Packages files by itself, as
well as a bunch of other source of information. But many of the most
recent developments have piggy-backed on UDD to retrieve the information
needed, leaving to UDD the responsibility of bringing all the information
in a single place.

This principle should be generalized to avoid duplication of work and to
make sure that all the important information are available in UDD.

But we must make sure that UDD won't become a bottleneck. Either because
we have a local (live?) replicate of the database, or because we have
ensured that our usage of UDD is limited to batch tasks that are not
on the critical path for all the real-time user requests.

### Using a modern framework for web development

DDPO is implemented in PHP. The PTS uses a mix of Perl, Python, XSLT and
shell scripts. While both works very well and are reliable, the diversity
of the tools and the fact that some are not widely known (e.g. XSLT)
seriously limit the set of contributors who are able to hack on all the
parts of the infrastructure.

With a modern framework for web development, we enlarge the set of people
who are able to help us develop and maintain this infrastructure. It also
offers us a proper separation between presentation and code, so that it's
easier to let web designers integrate this service with the general
look&feel of the various Debian websites. On top of this, we get a fully
internationalized website for free.

### API for data export

If the infrastructure is going to have a central role, there will be
requests to extract data out of the system. We should cater for this by
providing a public API (over HTTP) allowing to retrieve all the (public)
information in some standardized manner.

JSON seems to be a good option for data export. It allows other services
to reuse information from the DPMH, and it makes it easy for various
web services to retrieve the information dynamically via Javascript.

### Native support of packaging teams

Any Debian Developer must be able to create a "packaging team" in the
system. Each packaging team has a set of packages that it maintains (or
keeps an eye on). Anyone can "subscribe" to the team and gets (by default)
all correspondance of all packages associated to that team.

The team subscription can be tuned (much like the current PTS subscription)
to receive only a subset of the usual mails. A direct package subscription would
take precedence over a team subscription, thus allowing the user to
exclude some packages from its team subscription (or get more info for
some specific packages where they are particularly interested).


Implementation details
----------------------

Questions:

 * How do we store emails? For how long? (we store all mails except the BTS mails)
 * What language and web framework? (buxy's default choice: Python & Django)
 * How do we authenticate users? And for DD super-powers?
 * [To be completed]

Acronyms
--------

 * DPMH: Debian Package Maintenance Hub (this project)
 * DDPO: [Debian Developer's Packages Overview](http://qa.debian.org/developer.php)
 * PTS: [Package Tracking System](http://packages.qa.debian.org)
 * BTS: [Bug Tracking System](http://bugs.debian.org)
 * UDD: [Ultimate Debian Database](http://udd.debian.org)
 * DD: Debian Developer
 * WNPP: [Work Needing and Prospective Packages](http://www.debian.org/devel/wnpp/)
 * RFH: Request For Help
 * RFA: Request For Adoption
 * O: Orphaned
 * NMU: Non-Maintainer Upload

Feedback
--------

If you have comments about this proposal, please send them to
debian-qa@lists.debian.org.

Changes
-------

* 2011-01-13: Initial draft by Raphaël Hertzog.
* 2011-01-28: Integrate feedback from debian-qa@lists.debian.org.
1	hertzog	244	[[!meta title="DEP-2: Debian Package Maintenance Hub"]]
2
3			Title: Debian Package Maintenance Hub
4			DEP: 2
5			State: DRAFT
6	hertzog	247	Date: 2012-01-13
7	hertzog	244	Drivers: Raphael Hertzog <hertzog@debian.org>
8			URL: http://dep.debian.net/deps/dep2
9	plessy	284	Source: http://anonscm.debian.org/viewvc/dep/web/deps/dep2.mdwn
10	hertzog	244	Abstract:
11			Debian maintainers rely on a multitude of services (DDPO, PTS,
12			DDPO-by-mail, BTS, etc), and information sources, in order to do
13			their job. The flow of information varies greatly from case to case.
14			.
15			This proposal is about creating a central infrastructure
16			that would consolidate several of those services and that would
17			standardize the information flow.
18
19
20			Rationale
21			---------
22			This new package maintenance infrastructure is needed:
23
24			* to fix long standing problems;
25	hertzog	249	* to provide a clean base to implement new features:
26			* that will help maintainers do a better job;
27			* that will help packaging teams to organize themselves;
28			* that will help the QA team to ensure that all Debian packages are
29			well maintained.
30	hertzog	244
31			### Problems to solve
32
33			#### Maintainer vs Uploader
34
35			The flow of information is not the same depending on whether you're
36			listed in the Maintainer field (in which case most services mail you
37			directly) or not (in which case you're supposed to subscribe via the PTS
38			or via a dedicated mailing list). But the opposite is true as well, some
39			information is only available via the PTS and many maintainers have to
40			subscribe to the PTS while excluding almost everything just to get
41			the information they want.
42
43			See also [#507288](http://bugs.debian.org/507288) for some more
44			discussions on this topic.
45
46			This makes it very painful to change/switch the Maintainer field because
47	hertzog	250	people have to update their PTS subscriptions accordingly.
48	hertzog	244
49			#### Duplication of work / inconsistency between the DDPO and the PTS
50
51			The DDPO and the PTS are completely separate services. This leads to
52			duplication of work when a new information needs to be made available in
53			their respective interface. It can also lead to inconsistencies between
54			both services when bug occurs or when different choices are made.
55
56			#### Mailing lists as Maintainer
57
58			We often have mailing lists listed in the Maintainer field and it's not
59			clear who are the real package maintainers and how many of them there are.
60			The Uploaders field is often outdated, and/or is just a representation of
61			who worked last on the package instead of who feels responsible for the
62			package.
63
64			### Goals
65
66	hertzog	249	#### Provide a working replacement for DDPO/PTS
67	hertzog	244
68	hertzog	249	Since the service aims to merge the DDPO and the PTS, it must be a working
69			replacement of both services and its set of features must englobe the
70			features of the actual services.
71	hertzog	244
72	hertzog	250	#### Replace maintenance mailing lists
73	hertzog	249
74			Packaging teams often separate the mailing list that gets the bug traffic and
75			other notifications from their main discussion mailing list. This new
76			infrastructure should entirely replace the former kind of mailing lists.
77			Anybody receiving notifications and information directed to the package
78			maintainer should get them via this new infrastructure.
79
80			It allows us to know how many people are notified for a given problem.
81			If nobody is notified, the package is effectively orphaned. A more
82			interesting case to detect is when several persons are being notified but
83			all of them are MIA or marked as not being available for Debian (busy/in
84			vacation).
85
86	hertzog	252	#### Keep track of the list of maintainers
87
88			Packages using this infrastructure should automatically keep track
89			of the list of their maintainers because anyone subscribing to a package
90			must pick a "role":
91
92	hertzog	257	* "maintainer" (only DD can select this role, or DM for their own packages)
93			* "contributor"
94			* "follower" (aka lurker)
95
96	hertzog	250	#### Replace WNPP's RFH, RFA, O
97
98			Since the infrastructure already stores the information about who is
99			maintaining what, it only makes sense to extend it to provide the
100			list of orphaned packages (i.e. packages without maintainers).
101
102			RFH should be replaced by a system where the help request is better
103			formalized so that we can better direct new contributors in places
104			where their skills would be well used. Instead of just requesting
105			help, you could request:
106
107	hertzog	257	* a new maintainer (to replace you, i.e. RFA)
108			* a supplementary co-maintainer
109			* a bug triager
110			* a C/Python/Perl/… programmer (you should be able to choose the programming
111			language)
112
113	hertzog	250	Each request also documents whether there's an associated offer of
114			"mentorship" associated to the help request. Of course, there would
115			also be a free form description to give more details about what's
116			expected.
117
118			#### Replace the LowThresholdNmu list
119
120			The [LowThresholdNmu](http://wiki.debian.org/LowThresholdNmu) wiki page
121			is a hack to let people know when NMU are welcome and not frowned
122			upon. This information should be properly stored in the database
123			and it should be associated to each package.
124
125	hertzog	255	#### Provide better integration with the packaging VCS
126
127			A majority of packages are maintained in a VCS nowadays. It's thus
128			important to extract the relevant information and make it easily
129			accessible in the new interface.
130
131			Commits should appear in the usual activity stream of the package. We
132			should be able to instantly see whether the VCS contains pending changes
133			or not, or whether a new upstream version is in preparation there.
134
135	hertzog	251	#### Enable new interactions with maintainers
136	hertzog	250
137	hertzog	251	The central role of this new "communication infrastructure" makes it
138			possible to design new interactions with maintainers. Instead of being
139			only a source of information, the infrastructure could be used to query
140			package maintainers and/or let them provide supplementary information.
141
142			This could be used to improve the MIA tracking process.
143
144			This infrastructure would also be a more natural place to store the
145			"available for Debian work" boolean flag ("vacation") that's currently
146			never used because it's buried in db.debian.org and that it's not
147			practical to update it.
148
149			This infrastructure could also be used to let maintainers document the
150			responsibilities that they have agreed to endorse, and describe the
151			associated commitments. That way it would be easier to detect packages
152			that cannot be well maintained because the set of maintainers do not cover
153			all the tasks that must be assumed to have a properly maintained package.
154
155			#### Provide new services to packaging teams
156
157			Given that this infrastructure would have native support for packaging
158			teams, it would also be a good place to offer some standardized services
159			for them.
160
161			For example, one of the central tool for teams are their VCS and it can be
162			useful for teams to be able to monitor the state of their package in the
163			VCS. The [PET](http://pet.alioth.debian.org/) tool could be adapted,
164			integrated and made available to all teams by default.
165
166			#### Support alternate notification systems
167
168			Email is the only official media used to communicate information to
169			Debian package maintainers. If all the relevant mails are going through
170			a central service, it's possible to store those emails and to forward
171			the relevant information by other means (RSS, XMPP, IRC, etc.). Also
172			new maintainers can then have access to some historic information
173			that used to be private for no good reasons.
174
175
176	hertzog	249	High-level design of the new infrastructure
177			-------------------------------------------
178
179	hertzog	244	### Fixing the flow of information
180
181			In order to cleanly solve the problem of the information flow, and to get
182			rid of the hacks made everywhere to send a copy of the mails to the PTS,
183			packages would be (progressively) modified to indicate
184			“Maintainer: <source>@pkgmaint.debian.org” in their control file.
185	hertzog	254	The current content of the field would be moved to the “Uploaders” field.
186	hertzog	244
187			Until all packages have been converted, the PTS would forward copies of
188			the mails to ensure that the new infrastucture can still be used for all
189			packages (even those who have not been updated yet).
190
191			Using this intermediary address also solves the problem of maintainers who
192			orphan their packages and are still listed as maintainers in many released
193			packages.
194
195	hertzog	254	QUESTION: It would be cleaner if DAK, the BTS, and all relevant services,
196			could stop sending mails to the Maintainer field, and would instead always
197			send the mail to the new infrastructure. That way we wouldn't need to
198			change the Maintainer field and there would be no transition period.
199
200			The new infrastructure would then be configured with initial subscriptions
201			for emails listed in Maintainer fields (except for mailing lists, since
202			the infrastructure aims to replace them).
203
204	hertzog	249	### Leveraging UDD
205	hertzog	244
206	hertzog	249	At least the PTS has been parsing Sources/Packages files by itself, as
207			well as a bunch of other source of information. But many of the most
208			recent developments have piggy-backed on UDD to retrieve the information
209			needed, leaving to UDD the responsibility of bringing all the information
210			in a single place.
211	hertzog	244
212	hertzog	249	This principle should be generalized to avoid duplication of work and to
213			make sure that all the important information are available in UDD.
214	hertzog	244
215	hertzog	249	But we must make sure that UDD won't become a bottleneck. Either because
216			we have a local (live?) replicate of the database, or because we have
217			ensured that our usage of UDD is limited to batch tasks that are not
218			on the critical path for all the real-time user requests.
219
220			### Using a modern framework for web development
221
222			DDPO is implemented in PHP. The PTS uses a mix of Perl, Python, XSLT and
223	hertzog	250	shell scripts. While both works very well and are reliable, the diversity
224			of the tools and the fact that some are not widely known (e.g. XSLT)
225			seriously limit the set of contributors who are able to hack on all the
226			parts of the infrastructure.
227	hertzog	249
228	hertzog	250	With a modern framework for web development, we enlarge the set of people
229			who are able to help us develop and maintain this infrastructure. It also
230			offers us a proper separation between presentation and code, so that it's
231			easier to let web designers integrate this service with the general
232			look&feel of the various Debian websites. On top of this, we get a fully
233			internationalized website for free.
234
235	hertzog	249	### API for data export
236
237			If the infrastructure is going to have a central role, there will be
238			requests to extract data out of the system. We should cater for this by
239			providing a public API (over HTTP) allowing to retrieve all the (public)
240			information in some standardized manner.
241
242			JSON seems to be a good option for data export. It allows other services
243			to reuse information from the DPMH, and it makes it easy for various
244			web services to retrieve the information dynamically via Javascript.
245
246			### Native support of packaging teams
247
248			Any Debian Developer must be able to create a "packaging team" in the
249			system. Each packaging team has a set of packages that it maintains (or
250			keeps an eye on). Anyone can "subscribe" to the team and gets (by default)
251			all correspondance of all packages associated to that team.
252
253			The team subscription can be tuned (much like the current PTS subscription)
254			to receive only a subset of the usual mails. A direct package subscription would
255			take precedence over a team subscription, thus allowing the user to
256			exclude some packages from its team subscription (or get more info for
257			some specific packages where they are particularly interested).
258
259
260			Implementation details
261			----------------------
262
263			Questions:
264
265			* How do we store emails? For how long? (we store all mails except the BTS mails)
266			* What language and web framework? (buxy's default choice: Python & Django)
267			* How do we authenticate users? And for DD super-powers?
268			* [To be completed]
269
270	hertzog	244	Acronyms
271			--------
272
273	hertzog	249	* DPMH: Debian Package Maintenance Hub (this project)
274	hertzog	244	* DDPO: [Debian Developer's Packages Overview](http://qa.debian.org/developer.php)
275			* PTS: [Package Tracking System](http://packages.qa.debian.org)
276	hertzog	247	* BTS: [Bug Tracking System](http://bugs.debian.org)
277	hertzog	249	* UDD: [Ultimate Debian Database](http://udd.debian.org)
278			* DD: Debian Developer
279	hertzog	250	* WNPP: [Work Needing and Prospective Packages](http://www.debian.org/devel/wnpp/)
280			* RFH: Request For Help
281			* RFA: Request For Adoption
282			* O: Orphaned
283			* NMU: Non-Maintainer Upload
284	hertzog	244
285	hertzog	256	Feedback
286			--------
287
288			If you have comments about this proposal, please send them to
289			debian-qa@lists.debian.org.
290
291	hertzog	244	Changes
292			-------
293
294			* 2011-01-13: Initial draft by Raphaël Hertzog.
295	hertzog	250	* 2011-01-28: Integrate feedback from debian-qa@lists.debian.org.