Spam Filter

Google’s head of search spam Matt Cutts posted a longer video today answering what it is like to fight webspam at Google.

The questions posed by Brian Harnish of Westminster, California were:

What is a day in the life of a search spam team member like? What is the evolution of decisions in terms of how they decide which aspects of the search algorithm to update? Will certain things within the algorithm never be considered for removal?

In short, Google has both humans who fight spam manually and also engineers who write algorithms to fight spam. The human manual spam fighters handle reactive spam, for the most part, whereas the engineers focus on proactive spam fighting.

Matt explains the best spam fighters look for patterns and trends and try to figure out “what is the loophole they’re exploiting.” Then the engineer would work up an algorithm to expose and cover the loophole.

Most engineers spend their days coding and building algorithms. They often will build something, test it and refine. Then the algorithm might be tested in a live experiment where false positives and other issues might be discovered.

As Matt describes how engineers work, he makes it sound like an art. Where an engineer is not just looking to squash the loophole but creatively look for ways to catch the spam at a deeper level.

Often, the tasks set for the beginning of the day or the beginning of the quarter will change fast. What Google’s engineers set out to do may change based on a major issue or a big complaint from someone or somewhere, that complaint can come internally within Google or externally via a spam report, blogger or somewhere else. So it is a very “dynamic” space,” Cutts said but that also makes it interesting and “very fun and an intellectual problem.”

People will try to spam forever, as Matt has said before – so there is plenty of work and of course, job security. But engineers can work on anything, from existing algorithms to new ones, to making old algorithms faster or better, to building new algorithms for new issues.

“You never want to play whack a mole with a spammer,” Matt said but instead find way to plug a hole.