Wisdom of Crowds
There are few things like a scientific conference to make one feel as if he has been suddenly transported out of the ordinary world, into a fervor of activity focused on important problems facing mankind.
You knew we haven’t cured* cancer, right?
Maybe you did. But I bet you didn’t know that among the people who work on the complex complicated problem of cancer, some have given up all hope a cure will be discovered by one person or group. Unlike a hundred years ago when a mind-boggling problem could be solved by a lone frizzy-haired man who split his time between the patent office by day and lucubration over theory by night.
No. Today some people advocate we should turn to the wisdom of crowds.
Computational Models and Crowd-Sourcing Initiatives for Inferring Genetic Predictors of Cancer Phenotypes.
A catchy title of an interesting talk given in our session last week Friday. The basic idea was that if you can enroll a large enough number of groups of really smart people with different approaches to solving the same hard problem, then you just might. Take breast cancer for example. Determining a patient’s disease prognosis given her specific — “personalized” — molecular form of the disease (there are many), and then prescribing — “designing” — the best treatment so she won’t die from the disease, has been elusive. The literature is awash with failed attempts.
The presenter showed a rank ordered list of the performance of all the entries that had been submitted (~50) to a recent challenge in breast cancer. The challenge involved developing a predictive, computational model to separate a large group of breast cancer patients into two (or more) prognostic classes, given only the genotype (expression of ~20K genes) of the patients’ tumors, and suitable controls. Prediction accuracy of the models was evaluated on an independent “test” data set. In addition to gene expression, modelers were also permitted to use any other relevant bioinformatic data they deemed relevant. The group that submitted the model with the highest predictive accuracy on a “test” data set was awarded with a publication in a high impact scientific journal.
Interestingly, the second best performing model was also submitted by the winning group. So one of my takeaways was that it isn’t so much you need a crowd to find a solution to a hard problem, you just need to identify the right group in the first place. Although the presenter did mention to me over drinks later on that when aggregating the model predictions into one, performance was as good as the single best model, or possibly better, I don’t recall.
A question came up at the talk: What’s the incentive to participate in this kind of winner-take-all challenge?
It’s a good question, because developing a predictive model like this takes considerable time and effort, and if you win, sure, you get a decent publication, but if you don’t you not only get nothing (other than an acknowledgment of participation in the journal article), your scientific reputation potentially is tarnished. The presenter replied that yes, this was something they’d thought about, they’d considered spreading the wealth to all participants but couldn’t figure out how to do it since the award is not monetary. Even so, the disincentive is evidently not so large as to discourage participation — this year they received submissions from groups in over ten different countries.
* Unlike, say, smallpox, a disease that has been virtually eradicated by vaccination, “curing” cancer means stopping people from dying from their cancer, not from getting cancer in the first place. At least for somatic cancers, there is no chance we’ll ever be able to stop certain genes from mutating entirely.
