Simoiu, Sumanth, Mysore, and Goel’s “Studying the Wisdom of Crowds at Scale” investigates a familiar claim: groups often make better judgments than individuals.

The paper is useful because it does not treat that claim as folklore. It studies crowd performance through a large online experiment with 1,000 questions across 50 topical domains and more than 500,000 responses. The questions included numerical and categorical answers, and the domains spanned text, audio, video, and image-based tasks.

That scale matters. It lets the paper examine crowd judgment across different kinds of knowledge rather than relying on one narrow trivia-style setting.

What the paper shows

The paper reports three important findings.

First, crowds often outperform individuals. Aggregating many imperfect judgments can produce a better answer than relying on one person’s judgment.

Second, crowd performance is more consistent than individual performance. Individuals vary widely, but aggregation can smooth out some of that noise.

Third, social influence can damage crowd performance. When people see others’ answers, the group can begin to herd. Instead of adding independent information, participants may copy visible signals, and the crowd becomes less accurate.

That third finding is the most interesting part of the paper. The wisdom of crowds depends on the independence of the crowd. Once social influence becomes too strong, the mechanism that made the group useful begins to fail.

Why it matters

The paper connects directly to how modern digital systems work.

Recommendation systems, ratings, rankings, social media feeds, prediction markets, search results, and product reviews all rely on some form of aggregated human signal. These systems often assume that more participation means better information.

However, if everyone is reacting to everyone else, the signal is no longer independent. The system may look democratic while quietly amplifying early noise, popularity, status, or visibility.

That matters for AI too. AI systems increasingly learn from human feedback, user behavior, ratings, corrections, and engagement patterns. If those human signals are herded, biased, or shaped by the system itself, the model may learn the shape of the feedback loop rather than the truth of the underlying domain.

What I found especially useful

The experimental design is the paper’s biggest strength.

Using 1,000 questions across 50 domains gives the work more credibility than a small demonstration. The inclusion of different media types also helps. Crowd judgment is not one thing. Estimating a number, recognizing an image, predicting an outcome, and answering a knowledge question may all behave differently.

The paper also gives a practical warning: collective intelligence is conditional. It works best when the system preserves independent judgment and aggregates diverse evidence. It weakens when social pressure collapses that independence.

That is a good lesson for anyone designing systems around votes, likes, reviews, surveys, or human feedback.

What remains open

The paper shows that social influence can lead to herding, but it does not fully explain when that happens or why. Social influence is not always harmful. In some contexts, seeing others’ answers could help people correct mistakes or learn from better-informed participants.

The next question is therefore more specific:

  • When does social influence improve crowd performance?
  • When does it reduce performance?
  • Which domains are most vulnerable to herding?
  • Can interface design preserve independence while still allowing useful collaboration?
  • How should systems detect when crowd judgment has become an echo rather than evidence?

These questions matter because many real-world systems cannot simply remove social influence. The practical challenge is to design around it.

Bottom line

“Studying the Wisdom of Crowds at Scale” is valuable because it treats collective intelligence as an empirical question rather than a slogan.

The paper shows that crowds can be powerful without being automatically wise. The crowd works when it contributes independent information. Once the system turns participants into followers of each other’s signals, it stops adding information and starts amplifying whatever was already visible.

That condition matters for AI, social platforms, and any product that depends on aggregated human judgment.

Reference

Camelia Simoiu, Chiraag Sumanth, Alok Mysore, and Sharad Goel, “Studying the Wisdom of Crowds at Scale”, HCOMP 2019.