An open source internet stupidity filter

Filed under: Developer, Internet, Text, Blogging, Google, Open Source

An open source internet stupidity filter

by Ian Smith Oct 10th 2007

Stupid is as stupid does - an open source stupid filter

What is white and red and stupid all over? YouTube! Or at least that is that is the premise that the StupidFilter project is using to seed their self-training database. And really is there any form of stupidity with a more enduring appeal than a video discussion? Of course not!

The goal simple: build a stupid filter that works like a spam filter. Take a huge collection of stupid comments (225,000 to start), rate and organize them according to stupidness, and then teach the filter to recognize stupidity in the wild. Once the research is done a core engine will be released suitable for implementation in blogs, wikis, social networks, content management systems, and video sharing websites.

The current target release date for an alpha release is December 2007. After that the race begins: can the StupidFilter keep up with the diabolical adaptiveness of internet stupidity? Or will "stupiders" change their tactics and find ways around the filtering technology (by using complete words, for example)? Only time, and the efforts of a few brave programmers, will tell.

Tags: comment spam, CommentSpam, stupid filter, stupider, StupidFilter, youtube

Reader Comments (Page 1 of 1)

Christopher Finke said 5:23PM on 10-10-2007

My Firefox extension called YouTube Comment Snob achieves something like this by filtering YouTube comments based on spelling quality, capitalization, and punctuation usage: http://www.chrisfinke.com/addons/youtube-comment-snob/

Matt R said 8:43PM on 10-10-2007

This would be pleasantly ground breaking if they could bring it out of the virtual world.

Grant Robertson said 10:27PM on 10-10-2007

Bloody brilliant. Now the griefers and troglodytes will attach random text to their comments, much as the spammers did before them.

In seriousness though, this is actually a really neat idea. Why didn't I think of that?

John said 2:13AM on 10-11-2007

Ok, so lets put this to the test:

"OmG, 4th PoSt!!!! Dwnlaod Sqaud rulez!!! :) XoXXoxoX"

Thundercross said 4:29PM on 10-11-2007

I'm concerned with something like this being used as a censorship engine by flagging unpopular opinions as stupidity. afsjiaostnoruighoeigigjirgjoijoitae.

Michel said 9:50PM on 10-11-2007

I dont tink it will work as i know allot of people like to use abreviations and if the system starts taking those as stupidety well it will make many angry.

LEo said 4:21AM on 10-20-2007

Great! Nothing like a new censorship to cheer up my morning ( read it with a Jingjing voice: http://en.wikipedia.org/wiki/Jingjing_and_Chacha )

D said 3:08AM on 1-08-2008

Well yeah it might be misused as censorship...though I can't go 10 secs on the net without encountering a comment or post etc that blows me away with its force of stupidity. Maybe it's my own fault for frequenting those sites.

But any filter will have to be pretty sophisticated or else too severe a censor, as you can have a perfectly coherent and intelligent comment that uses abbreviations or colourful language in my opinion, so long as the underlying message is well thought out. It's just unfortunate the majority of what I see is riddled with grammatical errors (I mean basic ones - I don't claim to be an expert, but I did at least pay attention in bloody PRIMARY school) as well as illogical reasoning, prejudice or sheer ignorance.

I'm going to be labelled a prat for this long-winded post, but i had to get it off my chest

Julian said 6:44AM on 2-14-2008

Great stuff! It will take a lot of tweaking just like any spam filter. I thought it would be a good idea for Youtube to ask for your credit card (without charging it) in order to give you access to a non teenage section, hopefully with far less leetspeak and all that goes with it.

Add your comments

Your comments:

Remember me

E-Mail me when someone replies to this comment

Please keep your comments relevant to this blog entry. Email addresses are never displayed, but they are required to confirm your comments.

When you enter your name and email address, you'll be sent a link to confirm your comment, and a password. To leave another comment, just use that password.

To create a live link, simply type the URL (including http://) or email address and we will make it a live link for you. You can put up to 3 URLs in your comments. Line breaks and paragraphs are automatically converted — no need to use <p> or <br /> tags.

Featured Time Waster

Civiballs is a beautiful, soothing physics puzzle Time Waster

I have an absolute weakness for physics games, and while Civiballs isn't the strongest physics-based game, what it lacks in the physics department it makes up for a few times over in style and fun.

In Civiballs, you are presented with a few colored balls, and your goal is to get those balls into the same-colored urn on the level. The "civi" part of Civiballs is that there are 3 sets of levels to play, each representing a different civilization. While the civilization doesn't affect gameplay, the artwork for each level is beautifully themed to it's appropriate era.

To play the game, you are given only one tool - a sword with which to cut the chains that are holding the balls. The puzzle part of the game is in figuring out what order, and with what timing to cut each chain. Do it right, and all the right balls end up in the right urns, with no stray balls entering an urn (a no-no). Do it wrong, and you get to start over again.

Civiballs is not terribly deep on gameplay; the entire game can be completed in about 15 minutes. But if you enjoy this type of game, it will be a very enjoyable 15 minutes.

View more Time Wasters