Contents of /trunk/bits/list-update-200803

From: whoever send it
To: debian-devel-announce@lists.debian.org
Subject: Bits from the listmaster team

Hi everyone,

the listmaster team is constantly trying to improve the setup of our
listserver. Thus, quite a few things have happend since our last update
in September of last year. Here are some highlights:


lists.debian.org moved to a new hosting location
------------------------------------------------

lists.debian.org has been moved to man-da[2] to avoid several problems
with DNSBL pointing to our list server. We'd like to thank Brainfood
for hosting lists.debian.org and dealing with the insanity of spam
reporters for so long. After the move of the list service to the new
machine, we also decided to move the list archives to that machine
(which means the list archives are on the same machine as the MX, and
consequently fewer delays).

If you haven't already, please add lists.d.o's new ip, 82.195.75.100,
to your whitelists.


New list archive search engine
------------------------------
With the move of lists.debian.org to the new hosting, we took the
opportunity of deploying a new search[3] based on Xapian Omega. The
index comprises some 3.5 million messages, approximately 100k of which
are estimated to be spam.  In order to improve your search experience,
we have prepared the indexing software to benefit from our spam removal
plan (see below).
As before, searching by author and list is supporteed, the new search
should be more language- and encoding-aware.
Work is in progress to provide our adaptations to upstream and implement
improvements based on our experience and the hints we got from the
friendly people at Xapian.

New lists / Closed lists
------------------------
(zobel)


Config clean up
---------------
The Config Cleanup is another big project which seems to turn into an
ongoing task. Since the last update we decided to unify some global
files for all lists, and move all list specific config to extra files.
(This follows the layout the inventors of smartlist, our mailinglist
software, had in mind)

We also want to move some information like moderation-status or maximum
mailsize per message to a global file, which is also used by the
listarchive and some more informational or statistical tools.

To check if lists are configured correctly we subscribed an address to
all 182 mailinglists and checked back a month later for Ham/Spam-ratio,
and other anomalies. We found some wrong spam-rules, which led to some
false positives and other 'backdoors' which bypassed some of our
spamrules, which lead to false negatives.  We also found lists which are
supposed to carry only informational mails from an automatic system, so
we could tighten the rules, and on the other hand we could drop the
usual spamfilters for those lists, so distribution gets faster and we
need less CPU/Memory ressources to get one mail through.

We also implemented the usual 'Precedence' and 'List-*'-Headers on all
lists (we had some lists where those were missing) and Automatic
Responses, so we are now a little more net-friendly with our service.

While reviewing things we found that our bounce-handling had some
issues, see the next chapter for that.

Better bounce handling
----------------------
We checked our bounce-handling, because we got 500+ bounces for some
lists, and found that we didn't have a working bounce handling for some
lists (other-*, deity, *-digest, debian-private). There were also
problems in handling and recognizing mailadresses containing = or
!-characters.

Bounces of debian-private subscription are still manualy handed by the
listmasters, but we address these issues and forward such addresses to
da-manager@debian.org.

To address the other mailing lists we rewrote some parts of our
bounce-handler.

While analysing the bounces streaming in, we found that a lot of bounces
are caused by Content-Filters which reject listmail back to us (which
violates the RfC). Even worse: the majority of those are false
positives.

To let those people know we'll implement a notification-system, which
will notify users about bounces, and remind forcibly removed users about
their unsubscription.

This is a service for those people with a temporarily unavailable or
broken mailbox, so they see that they (or their provider) has a broken
Mail-setup or resubscribe back to all lists after their mailaddress is
functional again.  These notification will be sent out at a maximum of
once a week, up to a month after the last unsubscription happened.

Both notification systems are in testing now and will be activated
shortly after this mail.


List archive spam
-----------------
As avid followers of debian-project will know, we have implemented
support to weed spam out of the www list archives. While we want to
getting rid of as much spam as possible our paramount objective in this
effort is to preserve the integrity of the archive (e.g. keeping URLs
constant for past messages and avoiding removal of non-spam mail). This
means that the submissions we receive from users clicking on the
spam-report button of the list archive must be verified manually and
each nomination has to be checked by independently by several people.
There is a preliminary console-based program to aid the review process,
an epiphany plugin is being worked on by Cyril Brulebois.  To help out
or learn more please visit our wiki page[##z].

How to help
-----------
You can help us in a few important areas:

 * Spam rules -- If you notice spam getting through the spam filters,
   and have ideas for improving our filters, we accept patches to our
   rulessets, which are publicly available via svn.[##svn]

 * Encoding issues -- If you notice encoding problems after November
   2007 in the archive, please contact listarchives@lists.debian.org
   with a link to the problematic message and an explanation of the
   problem.

 * Avoid bouncing spam -- If you don't want your MTA to accept spam,
   please just discard it instead of 550'ing, at least when a message
   comes from liszt.debian.org

 * Troubleshooting -- If you notice a problem with a message that
   you've sent to a mailing list which hasn't arrived, please provide
   us with as much information as possible, including Date/Time (UTC),
   From, To, Message-Id, delivering IP, and the logfile entries from
   the delivering host.

[1] http://www.brainfood.com
[2] http://www.man-da.de
[3] http://lists.debian.org/search.html
[##z] http://wiki.debian.org/Teams/ListMaster/ListArchiveSpam
[##svn] svn://svn.debian.org/svn/pkg-listmaster/trunk/spamassassin_config
1	zobel	138	From: whoever send it
2			To: debian-devel-announce@lists.debian.org
3			Subject: Bits from the listmaster team
4
5	zobel	140	Hi everyone,
6	zobel	138
7	zobel	140	the listmaster team is constantly trying to improve the setup of our
8	zobel	150	listserver. Thus, quite a few things have happend since our last update
9			in September of last year. Here are some highlights:
10	zobel	140
11
12			lists.debian.org moved to a new hosting location
13			------------------------------------------------
14
15	cord	154	lists.debian.org has been moved to man-da[2] to avoid several problems
16	don	152	with DNSBL pointing to our list server. We'd like to thank Brainfood
17			for hosting lists.debian.org and dealing with the insanity of spam
18			reporters for so long. After the move of the list service to the new
19			machine, we also decided to move the list archives to that machine
20			(which means the list archives are on the same machine as the MX, and
21			consequently fewer delays).
22	zobel	140
23	don	152	If you haven't already, please add lists.d.o's new ip, 82.195.75.100,
24			to your whitelists.
25	cord	147
26	don	152
27	tviehmann	144	New list archive search engine
28	zobel	140	------------------------------
29	zobel	150	With the move of lists.debian.org to the new hosting, we took the
30			opportunity of deploying a new search[3] based on Xapian Omega. The
31			index comprises some 3.5 million messages, approximately 100k of which
32			are estimated to be spam. In order to improve your search experience,
33			we have prepared the indexing software to benefit from our spam removal
34			plan (see below).
35			As before, searching by author and list is supporteed, the new search
36			should be more language- and encoding-aware.
37	tviehmann	144	Work is in progress to provide our adaptations to upstream and implement
38	zobel	150	improvements based on our experience and the hints we got from the
39			friendly people at Xapian.
40	zobel	140
41			New lists / Closed lists
42			------------------------
43			(zobel)
44
45
46			Config clean up
47	cord	149	---------------
48	zobel	150	The Config Cleanup is another big project which seems to turn into an
49			ongoing task. Since the last update we decided to unify some global
50			files for all lists, and move all list specific config to extra files.
51			(This follows the layout the inventors of smartlist, our mailinglist
52			software, had in mind)
53	zobel	140
54	cord	142	We also want to move some information like moderation-status or maximum
55	zobel	150	mailsize per message to a global file, which is also used by the
56			listarchive and some more informational or statistical tools.
57	cord	142
58	zobel	150	To check if lists are configured correctly we subscribed an address to
59			all 182 mailinglists and checked back a month later for Ham/Spam-ratio,
60			and other anomalies. We found some wrong spam-rules, which led to some
61			false positives and other 'backdoors' which bypassed some of our
62			spamrules, which lead to false negatives. We also found lists which are
63			supposed to carry only informational mails from an automatic system, so
64			we could tighten the rules, and on the other hand we could drop the
65			usual spamfilters for those lists, so distribution gets faster and we
66			need less CPU/Memory ressources to get one mail through.
67	cord	142
68	zobel	150	We also implemented the usual 'Precedence' and 'List-*'-Headers on all
69			lists (we had some lists where those were missing) and Automatic
70			Responses, so we are now a little more net-friendly with our service.
71	cord	142
72	zobel	150	While reviewing things we found that our bounce-handling had some
73			issues, see the next chapter for that.
74	cord	142
75	zobel	140	Better bounce handling
76	cord	149	----------------------
77	zobel	150	We checked our bounce-handling, because we got 500+ bounces for some
78			lists, and found that we didn't have a working bounce handling for some
79			lists (other-, deity, -digest, debian-private). There were also
80			problems in handling and recognizing mailadresses containing = or
81			!-characters.
82	zobel	140
83	zobel	150	Bounces of debian-private subscription are still manualy handed by the
84			listmasters, but we address these issues and forward such addresses to
85			da-manager@debian.org.
86	cord	147
87	zobel	150	To address the other mailing lists we rewrote some parts of our
88			bounce-handler.
89	cord	142
90	zobel	150	While analysing the bounces streaming in, we found that a lot of bounces
91			are caused by Content-Filters which reject listmail back to us (which
92			violates the RfC). Even worse: the majority of those are false
93			positives.
94	cord	142
95	zobel	150	To let those people know we'll implement a notification-system, which
96			will notify users about bounces, and remind forcibly removed users about
97			their unsubscription.
98	cord	142
99	zobel	150	This is a service for those people with a temporarily unavailable or
100			broken mailbox, so they see that they (or their provider) has a broken
101			Mail-setup or resubscribe back to all lists after their mailaddress is
102			functional again. These notification will be sent out at a maximum of
103			once a week, up to a month after the last unsubscription happened.
104	cord	142
105	zobel	150	Both notification systems are in testing now and will be activated
106			shortly after this mail.
107	cord	142
108	zobel	150
109	zobel	140	List archive spam
110			-----------------
111	zobel	150	As avid followers of debian-project will know, we have implemented
112			support to weed spam out of the www list archives. While we want to
113			getting rid of as much spam as possible our paramount objective in this
114			effort is to preserve the integrity of the archive (e.g. keeping URLs
115			constant for past messages and avoiding removal of non-spam mail). This
116			means that the submissions we receive from users clicking on the
117			spam-report button of the list archive must be verified manually and
118			each nomination has to be checked by independently by several people.
119			There is a preliminary console-based program to aid the review process,
120			an epiphany plugin is being worked on by Cyril Brulebois. To help out
121			or learn more please visit our wiki page[##z].
122	zobel	140
123			How to help
124			-----------
125	don	148	You can help us in a few important areas:
126	zobel	140
127	don	148	* Spam rules -- If you notice spam getting through the spam filters,
128			and have ideas for improving our filters, we accept patches to our
129			rulessets, which are publicly available via svn.[##svn]
130	zobel	140
131	don	148	* Encoding issues -- If you notice encoding problems after November
132			2007 in the archive, please contact listarchives@lists.debian.org
133			with a link to the problematic message and an explanation of the
134			problem.
135	don	146
136	don	148	* Avoid bouncing spam -- If you don't want your MTA to accept spam,
137			please just discard it instead of 550'ing, at least when a message
138			comes from liszt.debian.org
139
140			* Troubleshooting -- If you notice a problem with a message that
141			you've sent to a mailing list which hasn't arrived, please provide
142			us with as much information as possible, including Date/Time (UTC),
143			From, To, Message-Id, delivering IP, and the logfile entries from
144			the delivering host.
145
146	zobel	140	[1] http://www.brainfood.com
147			[2] http://www.man-da.de
148	tviehmann	144	[3] http://lists.debian.org/search.html
149	tviehmann	145	[##z] http://wiki.debian.org/Teams/ListMaster/ListArchiveSpam
150	don	148	[##svn] svn://svn.debian.org/svn/pkg-listmaster/trunk/spamassassin_config