Spam and Dgroups


According to a survey from Internet security vendor Postini, spam traffic increased by 59% from September to November. That explained why Dgroups/Lyris did not function properly early this month. Usually there were 10,000 to 15,000 incoming messages each day before November, and it went up to 30,000 to 37,000. Such a volume killed Lyris. The poor Lyris used up all its energy to deal with incoming messages and slowed down other stuff.

Thanks for the help from ITMD, IDRC, we can temporarily put Dgroups server behind their high-performance, clustered anti-spam servers. The anti-spam server reduced traffic by 95%. Great Job! BTW, the anti-spam servers are dealing with 1.5 Millions message each day.

We were talking about to put an anti-spam server to protect Dgroups
server for a while. Since Lyris handled the traffic quite well and we
are always have too much things to work on, also considering the
negative impact of false positive, the plan did not get priority over
other stuff. It suddenly became urgent after the accident.

Since we don't have budget to equip high-end anti-spam servers, we need to come out a solution inexpensive but still works well. My colleague Graham setup a test anti-spam server with openbsd and spamd. We setup a honey spot called to attract spammers. Hope their harvester will find out this page ASAP and start to flood the spot with spam. I also spent some time on postfix+mailscanner.

There are 2 approaches we can take to fight against spammers. First we can develop our own anti-spam server. There are a number of open source, mature products such as Spamassassin, Dspam, mailscanner, etc. It requires in house resources to maintain the platform, keep it patched, update software components, etc. If you want to use enterprise class hardware such as HP DL-380 server, it will cost over $10,000. The return is that you have the control of everything, you can tailor it to fit your need the best.

You can also protect your server by acquire an anti-spam appliance. Basically they are customized version of first approach with commercial technical support. Usually these appliances are computers with everything is built in,come with a well designed WEB interface. Normally they are using content based filter to identify spam from legitimate messages. I found an interesting product from Deepsix technology. Their Spamwall DS-200 is a very small box with flash memory rather than harddisk. it identifies spammer solely according to characteristics of the sending server. According to a comparison from Windows Secret, it has the 0 false positive and very low (0.09%) false negative rate. It is much cheaper than popular anti-spam appliances (USD999 comparing to USD2,000 plus update service). Unfortunately, I contacted the vendor and was told the product is not sold in Canada. Also It can only handle up to 30,000 incoming messages daily but you can cluster multiple boxes to get greater capacity.

I also talked to Barracuda. Their anti-spam firewall products have won a lot of awards . I talked to their technical person and requested a evaluation unit. They allow me to return it within 30 days if I am not satisfied with the product. It will cost less than USD4,000 for the hardware and 3 years update subscription and 3 years instant replacement.

Personally I prefer the appliance approach which requires less effort to maintain the platform. I have no interest to spend too much my time to fight against spammers. I will leave it to anti-spam appliance vendors.

[Posted by Zhang Qu on Tuesday, November 28. 2006]