From: whoever send it To: debian-devel-announce@lists.debian.org Subject: Bits from the listmaster team Hi everyone, the listmaster team is constantly trying to improve the setup of our listserver. Thus, quite a few things have happend since our last update in September of last year. Here are some highlights: lists.debian.org moved to a new hosting location ------------------------------------------------ lists.debian.org has been moved away from Brainfood in succession of several problems with DNSBL pointing to our list server. After the move of the list service to the new machine (located at man-da[2]), we decided to also move the list archives to that machine (which means the list archives are on the same machine as the MX, and consequently fewer delays). If you didn't have already we ask you to add the new IP 82.195.75.100 to your whitelists. New list archive search engine ------------------------------ With the move of lists.debian.org to the new hosting, we took the opportunity of deploying a new search[3] based on Xapian Omega. The index comprises some 3.5 million messages, approximately 100k of which are estimated to be spam. In order to improve your search experience, we have prepared the indexing software to benefit from our spam removal plan (see below). As before, searching by author and list is supporteed, the new search should be more language- and encoding-aware. Work is in progress to provide our adaptations to upstream and implement improvements based on our experience and the hints we got from the friendly people at Xapian. New lists / Closed lists ------------------------ (zobel) Config clean up --------------- The Config Cleanup is another big project which seems to turn into an ongoing task. Since the last update we dicided to unify some global files for all lists, and move all list specific config to extra files. (This follows the layout the inventors of smartlist had in mind) We also want to move some information like moderation-status or maximum mailsize per message to a global file, which is also used by the listarchive and some more informational or statistical tools. To check if lists are configured correctly we subscribed an address to all 182 mailinglists and checked back a month later for Ham/Spam-ratio, and other anomalies. We found some wrong spam-rules, which led to some false positives and other 'backdoors' which bypassed some of our spamrules, which lead to false negatives. We also found lists which are supposed to carry only informational mails from an automatic system, so we could tighten the rules, and on the other hand we could drop the usual spamfilters for those lists, so distribution gets faster and we need less CPU/Memory ressources to get one mail through. We also implemented the usual 'Precedence' and 'List-*'-Headers on all lists (we had some lists where those were missing) and Automatic Responses, so we are now a little more net-friendly with our service. While reviewing things we found that our bounce-handling had some issues, see the next chapter for that. Better bounce handling ---------------------- We checked our bounce-handling, because we got 500+ bounces for some lists, and found that we didn't have a working bounce handling for some lists (other-*, deity, *-digest, debian-private). There were also problems in handling and recognizing mailadresses containing = or !-characters. To address these issues we rewrote some parts of our bounce-handler. While analysing the bounces streaming in, we found that a lot of bounces are caused by Content-Filters which reject listmail back to us (which violates the RfC). Even worse: the majority of those are false positives. To let those people know we implemented a notification-system, which will notify users about bounces at maximum of once a week. We also implemented a notification which informs forcibly unsubscribed users about the unsubscription. This is a service for those people with a temporarily unavailable or broken mailbox, so they can resubscribe back to all lists after their mailaddress is functional again. These notification will be sent out once a week, up to a month after the last unsubscription happened. Both notification systems are in alpha-testing now and will be activated in the next weeks. List archive spam ----------------- As avid followers of debian-project will know, we have implemented support to weed spam out of the www list archives. While we want to getting rid of as much spam as possible our paramount objective in this effort is to preserve the integrity of the archive (e.g. keeping URLs constant for past messages and avoiding removal of non-spam mail). This means that the submissions we receive from users clicking on the spam-report button of the list archive must be verified manually and each nomination has to be checked by independently by several people. There is a preliminary console-based program to aid the review process, an epiphany plugin is being worked on by Cyril Brulebois. To help out or learn more please visit our wiki page[##z]. How to help ----------- You can help us in a few important areas: * Spam rules -- If you notice spam getting through the spam filters, and have ideas for improving our filters, we accept patches to our rulessets, which are publicly available via svn.[##svn] * Encoding issues -- If you notice encoding problems after November 2007 in the archive, please contact listarchives@lists.debian.org with a link to the problematic message and an explanation of the problem. * Avoid bouncing spam -- If you don't want your MTA to accept spam, please just discard it instead of 550'ing, at least when a message comes from liszt.debian.org * Troubleshooting -- If you notice a problem with a message that you've sent to a mailing list which hasn't arrived, please provide us with as much information as possible, including Date/Time (UTC), From, To, Message-Id, delivering IP, and the logfile entries from the delivering host. [1] http://www.brainfood.com [2] http://www.man-da.de [3] http://lists.debian.org/search.html [##z] http://wiki.debian.org/Teams/ListMaster/ListArchiveSpam [##svn] svn://svn.debian.org/svn/pkg-listmaster/trunk/spamassassin_config