| 1 |
From: whoever send it
|
| 2 |
To: debian-devel-announce@lists.debian.org
|
| 3 |
Subject: Bits from the listmaster team
|
| 4 |
|
| 5 |
Hi everyone,
|
| 6 |
|
| 7 |
the listmaster team is constantly trying to improve the setup of our
|
| 8 |
listserver. Thus, quite a few things have happend since our last update
|
| 9 |
in September of last year. Here are some highlights:
|
| 10 |
|
| 11 |
|
| 12 |
lists.debian.org moved to a new hosting location
|
| 13 |
------------------------------------------------
|
| 14 |
|
| 15 |
lists.debian.org has been moved to man-da[2] to avoid several problems
|
| 16 |
with DNSBL pointing to our list server. We'd like to thank Brainfood
|
| 17 |
for hosting lists.debian.org and dealing with the insanity of spam
|
| 18 |
reporters for so long. After the move of the list service to the new
|
| 19 |
machine, we also decided to move the list archives to that machine
|
| 20 |
(which means the list archives are on the same machine as the MX, and
|
| 21 |
consequently suffer fewer delays).
|
| 22 |
|
| 23 |
If you haven't already, please add lists.d.o's new ip, 82.195.75.100,
|
| 24 |
to your whitelists.
|
| 25 |
|
| 26 |
|
| 27 |
New list archive search engine
|
| 28 |
------------------------------
|
| 29 |
With the move of lists.debian.org to the new hosting, we took the
|
| 30 |
opportunity of deploying a new search[3] based on Xapian Omega. The
|
| 31 |
index comprises some 3.5 million messages, approximately 100k of which
|
| 32 |
are estimated to be spam. In order to improve your search experience,
|
| 33 |
we have prepared the indexing software to benefit from our spam removal
|
| 34 |
plan (see below).
|
| 35 |
|
| 36 |
As before, searching by author and list is supported, but the new
|
| 37 |
search should be more language- and encoding-aware. Work is in
|
| 38 |
progress to provide our adaptations to upstream and implement
|
| 39 |
improvements based on our experience and the hints we got from the
|
| 40 |
friendly people at Xapian.
|
| 41 |
|
| 42 |
|
| 43 |
Config cleanup
|
| 44 |
--------------
|
| 45 |
The config cleanup is another big project which seems to turn into an
|
| 46 |
ongoing task. Since the last update we decided to unify some global
|
| 47 |
files for all lists, and move all list specific config to extra files.
|
| 48 |
(This follows the layout the inventors of smartlist, our mailinglist
|
| 49 |
software, had in mind.)
|
| 50 |
|
| 51 |
We also want to move some information like moderation status or maximum
|
| 52 |
mail size per message to a global file, which is also used by the
|
| 53 |
listarchive and some more informational or statistical tools.
|
| 54 |
|
| 55 |
To check if lists are configured correctly we subscribed an address to
|
| 56 |
all 182 mailinglists and checked back a month later for ham/spam-ratio,
|
| 57 |
and other anomalies. We found some wrong spam-rules, which led to some
|
| 58 |
false positives and other 'backdoors' which bypassed some of our
|
| 59 |
spamrules, which lead to false negatives. We also found lists which are
|
| 60 |
supposed to carry only informational mails from an automatic system, so
|
| 61 |
we could tighten the rules, and on the other hand we could drop the
|
| 62 |
usual spamfilters for those lists, so distribution gets faster and we
|
| 63 |
need less CPU/memory ressources to get one mail through.
|
| 64 |
|
| 65 |
We also implemented the usual 'Precedence' and 'List-*'-headers on all
|
| 66 |
lists (we had some lists where those were missing) and automatic
|
| 67 |
responses, so we are now a little more net-friendly with our service.
|
| 68 |
|
| 69 |
While reviewing things we found that our bounce handling had some
|
| 70 |
issues, see the next section for information about that.
|
| 71 |
|
| 72 |
Better bounce handling
|
| 73 |
----------------------
|
| 74 |
We checked our bounce handling because we have more than 500 bounces
|
| 75 |
for some lists, and in the process found that we didn't have working
|
| 76 |
bounce handling for other lists (other-*, deity, *-digest,
|
| 77 |
debian-private). There were also problems in handling and recognizing
|
| 78 |
mailadresses containing = or ! characters.
|
| 79 |
|
| 80 |
Bounces of debian-private subscription are still manualy handed by the
|
| 81 |
listmasters, but we now address these issues and forward such addresses to
|
| 82 |
da-manager@debian.org.
|
| 83 |
|
| 84 |
To address the other mailing lists we rewrote some parts of our
|
| 85 |
bounce handler.
|
| 86 |
|
| 87 |
While analysing the bounces streaming in, we found that a lot of bounces
|
| 88 |
are caused by content filters which reject listmail back to us (which
|
| 89 |
violates the RfC). Even worse: the majority of those are false
|
| 90 |
positives.
|
| 91 |
|
| 92 |
To let those people know we'll implement a notification system, which
|
| 93 |
will notify users about bounces, and remind forcibly removed users about
|
| 94 |
their unsubscription.
|
| 95 |
|
| 96 |
This is a service for those people with a temporarily unavailable or
|
| 97 |
broken mailbox, so they see that they (or their provider) has a broken
|
| 98 |
mail setup or resubscribe back to all lists after their mailaddress is
|
| 99 |
functional again. These notification will be sent out at a maximum of
|
| 100 |
once a week, up to a month after the last unsubscription happened.
|
| 101 |
|
| 102 |
Both notification systems are in testing now and will be activated
|
| 103 |
shortly after this mail.
|
| 104 |
|
| 105 |
|
| 106 |
List archive spam
|
| 107 |
-----------------
|
| 108 |
As avid followers of debian-project will know, we have implemented
|
| 109 |
support to weed spam out of the www list archives. While we want to
|
| 110 |
get rid of as much spam as possible our paramount objective in this
|
| 111 |
effort is to preserve the integrity of the archive (e.g. keeping URLs
|
| 112 |
constant for past messages and avoiding removal of non-spam mail). This
|
| 113 |
means that the submissions we receive from users clicking on the
|
| 114 |
spam-report button of the list archive must be verified manually and
|
| 115 |
each nomination has to be checked by independently by several people.
|
| 116 |
Some 1000 spam messages have been deleted from the archives of
|
| 117 |
debian-jave, debian-project, debian-python, and debian-vote.
|
| 118 |
To help out or learn more please visit our wiki page[4].
|
| 119 |
|
| 120 |
How to help
|
| 121 |
-----------
|
| 122 |
You can help us in a few important areas:
|
| 123 |
|
| 124 |
* Spam rules -- If you notice spam getting through the spam filters,
|
| 125 |
and have ideas for improving our filters, we accept patches to our
|
| 126 |
rulessets, which are publicly available via svn.[5]
|
| 127 |
|
| 128 |
* Encoding issues -- If you notice encoding problems of messages sent
|
| 129 |
after November 2007 in the archive, please contact
|
| 130 |
listmaster@lists.debian.org with a link to the problematic message
|
| 131 |
and an explanation of the problem.
|
| 132 |
|
| 133 |
* Avoid bouncing spam -- If you don't want your MTA to accept spam,
|
| 134 |
please just discard it instead of 550'ing, at least when a message
|
| 135 |
comes from liszt.debian.org
|
| 136 |
|
| 137 |
* Troubleshooting -- If you notice a problem with a message that
|
| 138 |
you've sent to a mailing list which hasn't arrived, please provide
|
| 139 |
us with as much information as possible, including Date/Time (UTC),
|
| 140 |
From, To, Message-Id, delivering IP, and the logfile entries from
|
| 141 |
the delivering host.
|
| 142 |
|
| 143 |
[1] http://www.brainfood.com
|
| 144 |
[2] http://www.man-da.de
|
| 145 |
[3] http://lists.debian.org/search.html
|
| 146 |
[4] http://wiki.debian.org/Teams/ListMaster/ListArchiveSpam
|
| 147 |
[5] svn://svn.debian.org/svn/pkg-listmaster/trunk/spamassassin_config
|
Parent Directory
| 
Revision Log