Rafe Needleman, in a recent review of Vanno, calls us a “crowdsourced Better Business Bureau 2.0″.   The Better Business Bureau part is pretty clear, but how exactly does crowdsourcing apply to what we’re doing? 

The term itself was first coined by Jeff Howe in 2006, but the concept has its origin in James Surowiecki’s 2004 book The Wisdom of Crowds.  Surowiecki’s book is of the genre that popularizes behavioral economics and psychology and includes Malcolm Gladwell’s bestsellers Blink and The Tipping Point and  Tim Hartford’s The Undercover Economist and The Logic of Life.  These books follow the proven bestseller formula  - overstate and oversimplify the case in the title and focus on situations where the idea seems to work.   Surowiecki’s is no exception, but to his credit, he spends almost half his book exploring cases where crowds aren’t so wise.  

So how do we know when to trust a crowd?  Surowiecki identifies four criteria - their opinions are diverse, independent and decentralized, and there is a systematic way to aggregate opinions and turn them into a collective judgment.   The absence of one or more of these can lead to mob behavior (think stock bubbles) or groupthink (think Bay of Pigs or WMD), where the crowd judgment is unreliable.

With that as context, let’s turn back to Web-based social media.  At first glance, social media and social news websites seem perfect for crowdsourcing, since the platforms offer an efficient way to aggregate opinion that is both diverse and decentralized.   But what about independence of opinion?    Given the social nature of the medium, the focus by users on gaining attention and stature, and the fact that the underlying sources of information that form opinion (print and online media) are driven by popularity, personality and an extremely short news cycle (technically speaking, they are highly correlated), can we be confident that opinions collected on social media sites are independent?   The short answer is no.

 But isn’t three out of four good enough?   What do we really lose by user opinions being correlated?   What we lose is the fundamental legitimacy of the statistical tools most sites have at their disposal to extract collective judgments from the articles, stories, votes and comments they receive.  Quite simply, if the data you start with aren’t independent, then the tools of classical statistics – from a simple regression analysis to big commercial software packages - don’t work.   It’s garbage in, garbage out.  

Social media sites aren’t, of course, the only ones who make these mistakes.  Smarter people than me have long argued that big statistical packages like SPSS are essentially weapons of mass distraction -  “black boxes” that researchers in all fields use with little or no thought at to whether their data (or their experimental design) are appropriate.  But social media sites are at particular risk here, given that their data are correlated to the bone, so to speak.

So what does all this mean for Vanno?   How can we you possibly ask you to trust our “crowdsourcing”, given all this?    Well, we like to think we’ve done social media crowsourcing the right way, by making lemonade (more reliable collective judgments) out of lemons (highly correlated opinions).

Instead of using classical (frequentist) statistics - giving equal and independent weight to each story, vote and comment, doing a regresson analysis and coming up with a collective judgment that is essentially a mean opinion and variance – we use a different paradigm.    Ours starts by establishing a belief base for the collective judgment, and then iterating that base as new evidence comes in.  In this approach, the correlations between opinions can actually add to the reliability of the collective judgment.  It’s called belief-revision, and is anchored in the Bayesian statistical methods that have revolutionized the way spam is filtered, credit card and insurance fraud detected, automobiles diagnosed and repaired and medical decisions made. 

We establish an initial company reputation score (belief base) when users submit the first few stories about a company - indicating in each case whether an aspect or aspects of the company’s reputation are strengthened or weakened – and voters respond by assessing the credibility of the initial claims.    As more stories, votes and comments come in, this new “evidence” is weighed against the belief base, and the scores adjust accordingly.    If some the stories and votes appear highly correlated – for example if a surge occurs right after a popular blogger posts a rant – we use that information to improve the accuracy of the reputation scores.   Were we using classical statistics, we would – technically speaking – have to throw out those correlated data. 

And that, we believe, is crowdsourcing done right.