Contents of /trunk/bits/list-update-200803

From: whoever send it
To: debian-devel-announce@lists.debian.org
Subject: Bits from the listmaster team

Hi everyone,

the listmaster team is constantly trying to improve the setup of our
listserver. Thus, quite a few things have happend since our last update
in September of last year. Here are some highlights:


lists.debian.org moved to a new hosting location
------------------------------------------------
lists.debian.org has been moved away from Brainfood in succession of
several problems with DNSBL pointing to our list server. After the move
of the list service to the new machine (located at man-da[2]), we
decided to also move the list archives to that machine (which means the
list archives are on the same machine as the MX, and consequently fewer
delays).

If you didn't have already we ask you to add the new IP 82.195.75.100 to
your whitelists.


New list archive search engine
------------------------------
With the move of lists.debian.org to the new hosting, we took the
opportunity of deploying a new search[3] based on Xapian Omega. The
index comprises some 3.5 million messages, approximately 100k of which
are estimated to be spam.  In order to improve your search experience,
we have prepared the indexing software to benefit from our spam removal
plan (see below).
As before, searching by author and list is supporteed, the new search
should be more language- and encoding-aware.
Work is in progress to provide our adaptations to upstream and implement
improvements based on our experience and the hints we got from the
friendly people at Xapian.

New lists / Closed lists
------------------------
(zobel)


Config clean up
---------------
The Config Cleanup is another big project which seems to turn into an
ongoing task. Since the last update we decided to unify some global
files for all lists, and move all list specific config to extra files.
(This follows the layout the inventors of smartlist, our mailinglist
software, had in mind)

We also want to move some information like moderation-status or maximum
mailsize per message to a global file, which is also used by the
listarchive and some more informational or statistical tools.

To check if lists are configured correctly we subscribed an address to
all 182 mailinglists and checked back a month later for Ham/Spam-ratio,
and other anomalies. We found some wrong spam-rules, which led to some
false positives and other 'backdoors' which bypassed some of our
spamrules, which lead to false negatives.  We also found lists which are
supposed to carry only informational mails from an automatic system, so
we could tighten the rules, and on the other hand we could drop the
usual spamfilters for those lists, so distribution gets faster and we
need less CPU/Memory ressources to get one mail through.

We also implemented the usual 'Precedence' and 'List-*'-Headers on all
lists (we had some lists where those were missing) and Automatic
Responses, so we are now a little more net-friendly with our service.

While reviewing things we found that our bounce-handling had some
issues, see the next chapter for that.

Better bounce handling
----------------------
We checked our bounce-handling, because we got 500+ bounces for some
lists, and found that we didn't have a working bounce handling for some
lists (other-*, deity, *-digest, debian-private). There were also
problems in handling and recognizing mailadresses containing = or
!-characters.

Bounces of debian-private subscription are still manualy handed by the
listmasters, but we address these issues and forward such addresses to
da-manager@debian.org.

To address the other mailing lists we rewrote some parts of our
bounce-handler.

While analysing the bounces streaming in, we found that a lot of bounces
are caused by Content-Filters which reject listmail back to us (which
violates the RfC). Even worse: the majority of those are false
positives.

To let those people know we'll implement a notification-system, which
will notify users about bounces, and remind forcibly removed users about
their unsubscription.

This is a service for those people with a temporarily unavailable or
broken mailbox, so they see that they (or their provider) has a broken
Mail-setup or resubscribe back to all lists after their mailaddress is
functional again.  These notification will be sent out at a maximum of
once a week, up to a month after the last unsubscription happened.

Both notification systems are in testing now and will be activated
shortly after this mail.


List archive spam
-----------------
As avid followers of debian-project will know, we have implemented
support to weed spam out of the www list archives. While we want to
getting rid of as much spam as possible our paramount objective in this
effort is to preserve the integrity of the archive (e.g. keeping URLs
constant for past messages and avoiding removal of non-spam mail). This
means that the submissions we receive from users clicking on the
spam-report button of the list archive must be verified manually and
each nomination has to be checked by independently by several people.
There is a preliminary console-based program to aid the review process,
an epiphany plugin is being worked on by Cyril Brulebois.  To help out
or learn more please visit our wiki page[##z].

How to help
-----------
You can help us in a few important areas:

 * Spam rules -- If you notice spam getting through the spam filters,
   and have ideas for improving our filters, we accept patches to our
   rulessets, which are publicly available via svn.[##svn]

 * Encoding issues -- If you notice encoding problems after November
   2007 in the archive, please contact listarchives@lists.debian.org
   with a link to the problematic message and an explanation of the
   problem.

 * Avoid bouncing spam -- If you don't want your MTA to accept spam,
   please just discard it instead of 550'ing, at least when a message
   comes from liszt.debian.org

 * Troubleshooting -- If you notice a problem with a message that
   you've sent to a mailing list which hasn't arrived, please provide
   us with as much information as possible, including Date/Time (UTC),
   From, To, Message-Id, delivering IP, and the logfile entries from
   the delivering host.

[1] http://www.brainfood.com
[2] http://www.man-da.de
[3] http://lists.debian.org/search.html
[##z] http://wiki.debian.org/Teams/ListMaster/ListArchiveSpam
[##svn] svn://svn.debian.org/svn/pkg-listmaster/trunk/spamassassin_config
1	zobel	138	From: whoever send it
2			To: debian-devel-announce@lists.debian.org
3			Subject: Bits from the listmaster team
4
5	zobel	140	Hi everyone,
6	zobel	138
7	zobel	140	the listmaster team is constantly trying to improve the setup of our
8	zobel	150	listserver. Thus, quite a few things have happend since our last update
9			in September of last year. Here are some highlights:
10	zobel	140
11
12			lists.debian.org moved to a new hosting location
13			------------------------------------------------
14	zobel	150	lists.debian.org has been moved away from Brainfood in succession of
15			several problems with DNSBL pointing to our list server. After the move
16			of the list service to the new machine (located at man-da[2]), we
17			decided to also move the list archives to that machine (which means the
18			list archives are on the same machine as the MX, and consequently fewer
19			delays).
20	zobel	140
21	zobel	150	If you didn't have already we ask you to add the new IP 82.195.75.100 to
22			your whitelists.
23	zobel	140
24	cord	147
25	tviehmann	144	New list archive search engine
26	zobel	140	------------------------------
27	zobel	150	With the move of lists.debian.org to the new hosting, we took the
28			opportunity of deploying a new search[3] based on Xapian Omega. The
29			index comprises some 3.5 million messages, approximately 100k of which
30			are estimated to be spam. In order to improve your search experience,
31			we have prepared the indexing software to benefit from our spam removal
32			plan (see below).
33			As before, searching by author and list is supporteed, the new search
34			should be more language- and encoding-aware.
35	tviehmann	144	Work is in progress to provide our adaptations to upstream and implement
36	zobel	150	improvements based on our experience and the hints we got from the
37			friendly people at Xapian.
38	zobel	140
39			New lists / Closed lists
40			------------------------
41			(zobel)
42
43
44			Config clean up
45	cord	149	---------------
46	zobel	150	The Config Cleanup is another big project which seems to turn into an
47			ongoing task. Since the last update we decided to unify some global
48			files for all lists, and move all list specific config to extra files.
49			(This follows the layout the inventors of smartlist, our mailinglist
50			software, had in mind)
51	zobel	140
52	cord	142	We also want to move some information like moderation-status or maximum
53	zobel	150	mailsize per message to a global file, which is also used by the
54			listarchive and some more informational or statistical tools.
55	cord	142
56	zobel	150	To check if lists are configured correctly we subscribed an address to
57			all 182 mailinglists and checked back a month later for Ham/Spam-ratio,
58			and other anomalies. We found some wrong spam-rules, which led to some
59			false positives and other 'backdoors' which bypassed some of our
60			spamrules, which lead to false negatives. We also found lists which are
61			supposed to carry only informational mails from an automatic system, so
62			we could tighten the rules, and on the other hand we could drop the
63			usual spamfilters for those lists, so distribution gets faster and we
64			need less CPU/Memory ressources to get one mail through.
65	cord	142
66	zobel	150	We also implemented the usual 'Precedence' and 'List-*'-Headers on all
67			lists (we had some lists where those were missing) and Automatic
68			Responses, so we are now a little more net-friendly with our service.
69	cord	142
70	zobel	150	While reviewing things we found that our bounce-handling had some
71			issues, see the next chapter for that.
72	cord	142
73	zobel	140	Better bounce handling
74	cord	149	----------------------
75	zobel	150	We checked our bounce-handling, because we got 500+ bounces for some
76			lists, and found that we didn't have a working bounce handling for some
77			lists (other-, deity, -digest, debian-private). There were also
78			problems in handling and recognizing mailadresses containing = or
79			!-characters.
80	zobel	140
81	zobel	150	Bounces of debian-private subscription are still manualy handed by the
82			listmasters, but we address these issues and forward such addresses to
83			da-manager@debian.org.
84	cord	147
85	zobel	150	To address the other mailing lists we rewrote some parts of our
86			bounce-handler.
87	cord	142
88	zobel	150	While analysing the bounces streaming in, we found that a lot of bounces
89			are caused by Content-Filters which reject listmail back to us (which
90			violates the RfC). Even worse: the majority of those are false
91			positives.
92	cord	142
93	zobel	150	To let those people know we'll implement a notification-system, which
94			will notify users about bounces, and remind forcibly removed users about
95			their unsubscription.
96	cord	142
97	zobel	150	This is a service for those people with a temporarily unavailable or
98			broken mailbox, so they see that they (or their provider) has a broken
99			Mail-setup or resubscribe back to all lists after their mailaddress is
100			functional again. These notification will be sent out at a maximum of
101			once a week, up to a month after the last unsubscription happened.
102	cord	142
103	zobel	150	Both notification systems are in testing now and will be activated
104			shortly after this mail.
105	cord	142
106	zobel	150
107	zobel	140	List archive spam
108			-----------------
109	zobel	150	As avid followers of debian-project will know, we have implemented
110			support to weed spam out of the www list archives. While we want to
111			getting rid of as much spam as possible our paramount objective in this
112			effort is to preserve the integrity of the archive (e.g. keeping URLs
113			constant for past messages and avoiding removal of non-spam mail). This
114			means that the submissions we receive from users clicking on the
115			spam-report button of the list archive must be verified manually and
116			each nomination has to be checked by independently by several people.
117			There is a preliminary console-based program to aid the review process,
118			an epiphany plugin is being worked on by Cyril Brulebois. To help out
119			or learn more please visit our wiki page[##z].
120	zobel	140
121			How to help
122			-----------
123	don	148	You can help us in a few important areas:
124	zobel	140
125	don	148	* Spam rules -- If you notice spam getting through the spam filters,
126			and have ideas for improving our filters, we accept patches to our
127			rulessets, which are publicly available via svn.[##svn]
128	zobel	140
129	don	148	* Encoding issues -- If you notice encoding problems after November
130			2007 in the archive, please contact listarchives@lists.debian.org
131			with a link to the problematic message and an explanation of the
132			problem.
133	don	146
134	don	148	* Avoid bouncing spam -- If you don't want your MTA to accept spam,
135			please just discard it instead of 550'ing, at least when a message
136			comes from liszt.debian.org
137
138			* Troubleshooting -- If you notice a problem with a message that
139			you've sent to a mailing list which hasn't arrived, please provide
140			us with as much information as possible, including Date/Time (UTC),
141			From, To, Message-Id, delivering IP, and the logfile entries from
142			the delivering host.
143
144	zobel	140	[1] http://www.brainfood.com
145			[2] http://www.man-da.de
146	tviehmann	144	[3] http://lists.debian.org/search.html
147	tviehmann	145	[##z] http://wiki.debian.org/Teams/ListMaster/ListArchiveSpam
148	don	148	[##svn] svn://svn.debian.org/svn/pkg-listmaster/trunk/spamassassin_config