Tony Dye

ROI on Spam Protection

Phil James did a post on the ROI of Postini for his church.  That was pretty interesting, so I decided to do my own calculations.  WOW!

Instead of Postini, we use Katharion, supplemented by a Barracuda, but the analysis is similar.  Each day, we block approximately 27,000 spam messages!  That’s 9,855,000 per year.  Even at only 3 seconds each, that’s around 29 MILLION seconds, 492 THOUSAND minutes, 8000 HOURS, over 1000 [working] days, or roughly 4 YEARS of staff time (FIVE at 200 work days/year)!  What’s the value of 4 years?  And…is 3 seconds a realistic calculation for some of our biggest spam recipients?

Factor in anti-phishing and anti-hoax, as well as anti-virus, and that protection value goes even higher…

Anti-Spam: the Worst Things

I've been in the anti-spam battle for a long time.  I was recently talking with another IT manager about spam and how he's using his Barracuda.  We had a bit of dialog about "what's the worst thing" regarding spam.  Having spam get through is bad.  But it's not the worst thing.  Here are what I think are the three worst things in fighting spam:

The third worst thing: having spam get through to users.  Of course, this is what we're trying to fight, but having spam get through, although bad, isn't the absolutely worse thing that can happen.

The second worst thing that can happen is blocking a good email.  False positives are the challenge in anti-spam solutions.  I would much rather have hundreds of spam messages get through than to block a single good message.  But blocking a good message isn't really the absolutely worst thing that can happen.

The worst thing: The absolutely worst thing that can happen in your anti-spam solution is to block a good email and not let anybody know about it!  Anti-spam solutions should always generate an NDR such that a legitimate sender can know their message didn't get through. (Of course, we know many legitimate users don't read nor understand NDRs, so there's still an issue)  A really good anti-spam solution should not only generate an NDR, but that NDR should have an "escape clause" in it that gives that legitimate user a special way to get through the anti-spam solution, if they take some reasonable steps.

Does your anti-spam solution help ensure that legitimate mail gets through?  Does it actually help?

Even Happier with Katharion

I reported a few days ago that we were surprised at the "easy" things Katharion was missing each day.  Well, I owe a big apology to the folks at Katharion.  They were NOT missing those easy things.  But...they were processing a domain whitelist rule that somebody (not me, thankfully!) had put in place.  One little change and now the Barracuda isn't having much to do.  We're liking this more and more each day!

Katharion vs. Barracuda

In our war on spam, we've done a lot of different things through the years.  Most recently, we've started using Katharion as a front end anti-spam service, with our existing Barracuda as a second step in the pipeline.  We added Katharion because the Barracuda was letting too much through without considerable management.  Here's a snapshot from the Barracuda that shows the effectiveness of Katharion:

Notes:

1. Prior to 3/3, Katharion was operating in a minimal mode, mostly just passing mail through, but still knocking out a bit of spam.
2. Late in the day of 3/2 (effectively 3/3), we turned on LDAP filtering.  The drop in volume is quite noticeable.
3. Late in the day of 3/9 (effectively 3/10), we raised the aggressiveness of the Katharion filtering.  Again, the drop in volume is obvious.
4. We've continued to do fine tuning since then, but no more major changes in statistics.

Here's what I find most interesting.  We went to Katharion because the Barracuda was letting some fairly consistent types of spam through.  Katharion blocked those as soon as we cranked up the settings.  However, even with Katharion set at the most aggressive setting, it lets through some 100-200 messages a day that the Barracuda is able to fairly easily catch!  My guess is that there are still something like 20-50 messages a day that get through both filters, but I don't have good statistics here since we only know what our users tell us about.

Here's a daily, hour-by-hour view, from the Barracuda of current trends:

The traffic volume corresponds to working hours reasonably well.  Notice that almost every hour, some 2 or 5 or 10 or 20 messages come through that are recognized as spam, and there are very few false positives here.

I guess I could conclude that Barracuda and Katharion each have some issues to work on to improve the quality of what they do.  Or...maybe it's just that catching ALL the spam is really hard!  Right now, Katharion is letting through around 150-200 spam messages a day, vs. a volume of good messages of 3000-4000.  Is a 5% failure rate acceptable?  (Of course, using the double filtering, our real failure rate is more like 1/2%)  How effective is your anti-spam solution?  And the bigger question: how do you know?

Katharion "Fully Engaged"

Last week we made two big changes with Katharion.  First, we turned on full LDAP filtering.  Then, a day later, we cranked up the default setting to be much more aggressive on spam.  What a difference!  We're in the interesting/fortunate situation of having two anti-spam processes, Katharion first, then our Barracuda behind that, so we get a nice look at the results from Katharion, but also a second chance to filter out some things.  Here's a view of what the Barracuda is seeing from Katharion:
 

Late in the day of March 2 is when full LDAP was turned on, and instantly the Bad Recipients went away, as expected.  Late in the day March 9 is when we cranked up the Katharion anti-spam settings, and not surprisingly, spam just about disappeared...

What's most interesting is the small bit of spam that Katharion does let slip through, that we can fairly easily catch in the Barracuda.  We'll have to do some more analysis there, and perhaps give some good feedback to the Katharion people.

Katharion LDAP

At long last, we have LDAP filtering enabled with Katharion.  This was complicated because of the large number of distribution lists and email-enabled Exchange Public Folders we have.  If we were to present Katharion with our entire list of valid email addresses, we'd be opening ourselves up to a rather hefty bill for "accounts" that really aren't for outside use.  John and his crew got it figured out.  By creating another OU and adding a bit of descriptive info, they were able to create an LDAP query that presented the valid addresses to Katharion.  Finally, the trigger was pulled.  Think it made a difference?  Here are some statistics, as seen by our Barracuda:

Is it pretty obvious what time we turned on LDAP filtering?

Previous Katharion Posts:
A Full Day of Katharion
Katharion Update
Katharion's Improved Junk Mail Report

Katharion's Improved Junk Mail Report

Katharion has made a nice little improvement in their anti-spam service.  Katharion sends each user a daily "junk mail" report showing all the messages that were blocked, and allowing the user to deliver any of those messages.  Until about a week ago, that report was just one [sometimes long] list of all the messages blocked.  For users who have 50 or more messages blocked a day, it is quite a pain to read through that list and see if you care about anything.

With the latest updates, the report is now split into two parts.  A "probably" spam list, which has moderate risk of not being spam, and a "defionitely" spam list, which is hardly worth scanning.  By splitting this junk list in two like this, it makes the daily user processing much faster and simpler.

Nice work Katharion!

Katharion Update

We've been running Katharion for a bit over two weeks now.  Has it made a difference?  Most of our users probably haven't noticed much yet, but take a look at this graph from our Barracuda. 

It's pretty obvious the day we turned on Katharion.  Next up is to turn on LDAP processing, which will clean up things like those big spikes last week, then we'll start taking our spammiest users to higher settings.  It will be interesting to do another snapshot in about 2 more weeks.

A Full Day of Katharion

We've been running Katharion for a few days now.  We're still set at the lowest possible anti-spam setting, so our Barracuda is still doing a little work, but what a difference.  So take a look at these statistics from our Barracuda.  Pretty obvious when Katharion went into operation?

We get a few simple email reports each day that give a quick indication of what's going on. Here's the overall summary report for all users:

Here's my personal summary:

Then I also get a daily report showing *exactly* what was blocked, giving me a chance to release it:

Included in that daily report is a list of all the viruses, including phishing scams, that were blocked:

Yup, I think I'm liking this...

First Steps With Katharion

As happy as we've been with our two Barracudas, over the last few months they just haven't been doing the job and we've been spending way too much time managing spam on a daily basis.  So, having listened to Jason speak so well of Postini, we were ready to give it a try.  As we were making the calls, we also started hearing about Katharion, which really didn't interest us at first.  Well, short version: Postini wasn't real responsive to our calls, so we called Katharion and they were very happy to work with us.  A little setup and configuration, and finally a switch of the MX records.  Here's a quick snapshot of our inbound statistics from the Barracuda.  See if you can tell when mail started flowing through Katharion?
Unfortunately, we'd forgotten to turn off Rate Control in the Barracuda, so we were deferring good (and bad) mail for a bit.  Fortunately, that was an easy fix.  Initially, we were just passing mail through Katharion, not really doing any spam filtering.

You can see pretty clearly when we switched to *Minimal* Katharion spam filtering.  Quite a difference in mail volume, or should I say, quite a reduction in BAD mail volume.  With this minimal setting, the first thing I noticed is how we immediately stopped seeing the invalid email addresses.  What's so interesting is that we aren't doing any LDAP processing yet.  That reduction was simply because Katharion is recognizing the fast majority of those as spam.  That's nice.  (The Barracuda did that too, but only after consuming some of our Internet bandwidth with the junk mail).  So, we're only a few hours into this experiment, but so far, it looks good.