Tech lessons learned from the wisdom of crowds

Declan McCullagh of CNet.com discusses how Silicon Valley is learning to love prediction markets.

********************

SUNNYVALE, Calif.--In a 1945 paper, the great Austrian economist F.A. Hayek described how prices set by a free market are really "a mechanism for communicating information" about the probability of future events.

If a war in the Middle East is seen as likely, for instance, oil prices will probably increase. Hayek's insight showed that the results can be surprisingly accurate, as long as enough people are allowed to wager real money on the outcome.
Prediction markets

Now, technology firms are using a modern twist on this idea, called prediction markets, as a way to save money, harness the distributed knowledge of their rank-and-file employees, and even answer questions like: When will this software ship? And what will memory prices be like in a few months?

At a micro-conference that Yahoo convened at its headquarters here on Wednesday evening, representatives from Google, Yahoo, Microsoft and Hewlett-Packard described their experiments with prediction markets. "The really valuable knowledge is in the organization as a whole," said moderator James Surowiecki, author of the book The Wisdom of Crowds.

The companies' experiences went as follows:

Google: It's no secret that Google is using prediction markets internally: Bo Cowgill, a Google project manager, described it in a blog post last year.

Cowgill suggested that companies use reputation systems that encourage employee participation by showing whether someone is scoring better or worse in the prediction realm than his colleagues. "You can have winners in each department," Cowgill said, adding that he'd like Google's employee directory to reveal each person's accuracy.

Google offers cash prizes and T-shirts to employees who score well, and the winners are calculated quarterly, Cowgill said. But he still stressed reputation systems as a better method: "In order to make it attractive in a forward sense you have to spend more and more money or the laws are going to have to change. I don't think either is likely to happen anytime soon."

HP: HP Labs has created betting software called BRAIN that's designed for employees and managers; pharmaceutical giant Pfizer will use it starting next year. The unique feature is that BRAIN assigns people a profile based on how risk-averse they are.

To test BRAIN, HP compared it to existing ways of forecasting memory prices--a key figure since memory represents up to 10 percent of the cost of a computer and HP spends billions of dollars on it a year. Instead of a 4 percent error rate for the existing method of forecasting DDR and DDR2 512MB memory prices, the prediction market had a 2.5 percent error rate, or a 37 percent improvement. A second trial to forecast HP Services operating profits yielded similar results.

"We are able to pay real cash money to people without running afoul" of the Commodity Futures Trading Commission said HP's Leslie Fine, because the betting market gives participants only rewards and does not let them lose money.

Yahoo: David Pennock, a principal research scientist at Yahoo Research, said the company has created a currency called a Yootle. It's described as a "scorekeeping system for favors owed."

Pennock offered as an example a programmer offering to write a piece of code for a few Yootles. Or, when organizing a dinner outing, one employee could use an internal SMS tool to bid 2 Yootles for Italian and 4 Yootles for Mexican. "If you don't get to go to the restaurant you want to, you get compensation" in Yootles, he said.

Related to Yootles is Yahoo Research's experiment with a fantasy prediction market for technology called the Tech Buzz Game. It's a modified version of software licensed from NewsFutures in conjunction with O'Reilly Media and features topics like Atlantic hurricanes and portable media devices. Winners are those who predict how popular a topic will be on Yahoo Search.

Microsoft: When Todd Proebsting, director of Microsoft's Center for Software Excellence, tested a prediction market internally, managers quickly gave it their blessing.

The goal: to have 25 members of a development team predict when a Microsoft product would ship (this was an internal product, not one sold externally). The prediction market was set up in August 2004, and the product that "had been in the works for a long time" was scheduled to ship in November 2004. Each "trader" received $50 in their account to start with, and was told that the more accurate their prediction, the more money they would make. The market opened with an initial price of on-time delivery set to 16 2/3 cents.

"The price of 'before November' dropped to zero right away," Proebsting said. "The price of 'on time' in about two to three minutes dropped to 2.3 cents on the dollar." Translated, that's more than 30-to-1 odds against on-time delivery.

Then the woman who was responsible for scheduling started trying to convince her colleagues who were buying and selling future delivery dates. "She was able to talk (on-time delivery) up to around 3 cents," Proebsting said. "People really enjoyed moving the price...They loved this."

"The next day the director comes into my office and said, 'What have you done?'" Proebsting said. But further investigation showed that the product actually was behind schedule, even though nobody was telling management, and it eventually shipped in February.

Policy analysis markets: Chris Hibbert is an inventor and programmer who's writing open-source software to build a prediction market. It's called Zocalo.

Prediction software should work with play money, real money and an internal corporate currency that might be reputation-based, Hibbert said.

Most existing prediction markets like the Hollywood Stock Exchange and Tradesports.com focus too much on topics like sports and movie success, when policy and political questions can be more interesting, Hibbert said.

Robin Hanson, an associate professor of economics at George Mason University, is the closest that prediction markets have to a founding father. But he became a politically controversial one when his idea of a policy analysis market that could, for instance, help to predict terrorist attacks was savaged by Democratic senators eager to take a swipe at the Republican administration that funded it.

"The example that you provide in your report would let participants gamble on the question, 'Will terrorists attack Israel with bioweapons in the next year?'" Sens. Ron Wyden, D-Ore., and Byron Dorgan, D-N.D., wrote to the Defense Advanced Research Projects Agency in 2003. "Surely, such a threat should be met with intelligence gathering of the highest quality--not by putting the question to individuals betting on an Internet Web site."

On Wednesday, Hanson quipped that the flap wasn't the publicity "we would have asked for in starting an industry." A paper he wrote, though, showed that more informed journalists and more recent news articles were more likely to give a positive impression of policy analysis markets.

Hanson said that conditional probabilities were worthy of more experimentation: If a company switched ad agencies, would it increase revenue? If the U.S. provides cash subsidies to Jordan, will that nation have a better economy or a worse one?

"The market is telling you the consequences of your actions," Hanson said. "If you do this, what are the consequences?"