Explaining the better wisdom of crowds
Exploring the state space.
Mark Thoma points us to MIT News about improving the wisdom of crowds:
Their method ... uses a technique the researchers call the “surprisingly popular” algorithm to better extract correct answers from large groups of people. ...
The new method is simple. For a given question, people are asked two things: What they think the right answer is, and what they think popular opinion will be. The variation between the two aggregate responses indicates the correct answer. ...
“The argument in this paper, in a very rough sense, is that people who expect to be in the minority deserve some extra attention,” ...
The interesting thing here is that this method essentially creates a measure of the degree of exploration of the "answer state space" (in economics: the opportunity set) as well as indicate possible strong "correlations". These are exactly the kinds of things you need to know in order to determine whether e.g. prediction markets are working.
Agents fully exploring the state space is critical to ideal information transfer in prediction markets. Correlations (agents cluster in one answer state) are one way to reduce information entropy; lack of exploration (no agents select particular states) is another.
In the example in the article, they ask the question "Is Philadelphia the capital of Pennsylvania?"
In a sense, the question itself sets up a correlation by anchoring Philadelphia in respondents' minds. Those that answer "no" indicate they have explored the state space (the set of all cities in Pennsylvania) -- they've considered other cities that might be the answer. Those that say others will answer "yes" give a measure of either a lack of exploration or a correlation. The people who expect to be in the minority deserve extra attention because they have more likely explored the state space.
Update 3 February 2017
Noah Smith has a Bloomberg View article about this result. What is interesting is that both his theories can be unified under a lack of state space exploration. Dunning-Kruger [pdf] is at its heart a lack of knowing there exist parts of the state space where you are wrong. You think most people will answer Philadelphia because you haven't considered that other people might consider that Harrisburg is the capital of Pennsylvania. As a non-expert, you probably haven't explored the same state space experts have. Noah's second theory is herd behavior; this is exactly the correlation I mention above.
...
Here's some more reading and background on information equilibrium, prediction markets, and state space exploration (what Jaynes called dither) at the links below. A slide package is here (the first few slides about Becker's "irrational" agents are relevant).
Thinking positive is thinking different
Corporate prediction markets aggregate random behavior
Jaynes on entropy in economics
...
PS I always get worried when information equilibrium "explains everything". It is true that this is an indicator that the theory is correct. However, it is also an indicator of an overactive left brain interpreter.