CAMP (Critical Assessment of Mutant Prediction)

This is the project web to document planning for the Critical Assessment of Mutant Prediction idea (see Stoltzfus_CAMP.pdf), including an initial pilot project to assess feasibility.

Plan of action and "to do" items

• project planning, feedback and revision phase (document this on twiki)
  • prepare easy-to-read synopsis describing CAMP and the yeast knockout feasibility project (4 hrs) (done: March)
  • set up a public web site or twiki web describing the project (used DREAM discussion board)
  • recruit partners, including at least 6 potential prediction teams (email, phone calls) (?? 6 hrs)
  • get feedback on plan (e.g., teleconference, online discussion board), analyze (6 hrs)
  • identify any need for initial results or analysis of feasibility
• project funding phase
  • identify possible sources of support
    • NIST internal -- not available for 2008 due to budget constraints
    • NIH R21
    • donation of arrays from manufacturer?
  • make detailed plan, estimate time and expenses
  • submit proposal
• project execution
  • experimental phase
  • prediction phase
  • analysis phase
• followup with major CAMP project

Project overview

Is systems biology a predictive science that will revolutionize biology, or just the latest flavor of intellectual catnip? To find out, we propose a critical assessment of systems biology based on the notion that, in various practical contexts, understanding a system means either i) being able to predict the response to perturbation, or the inverse problem of ii) being able to infer a perturbation based on an observed response. One particularly important kind of perturbation in biology is mutation, thus as a Grand Challenge we propose a Critical Assessment of Mutant Prediction (CAMP) in which systems modelers either will predict the effects of mutation on a system, or infer a mutation from measurements on the perturbed system.

For more information, see Stoltzfus_CAMP.pdf, a short presentation describing the CAMP idea

Feasibility study: Yeast Knockout prediction from microarray data

To evaluate the feasibility of CAMP, we will characterize gene expression patterns in a few dozen randomly chosen yeast knock-out mutants (using microarray technology), then release the anonymous microarray data and challenge systems modelers to infer the underlying mutation.

Strategy

The flow of operations is outlined in two flowcharts

• CAMP_yeast_pilot1.pdf shows the major phases of data collection, data preparation, community prediction, and analysis
• CAMP_yeast_pilot2.pdf indicates some critical design issues to be resolved

Literature survey

What kinds of systems-level models and analysis of yeast regulation have been done?

• Schlitt and Brazma (2007)
• ways to get data to fill in network model
  • network topology (edge has no sign, is not quantitative)
    • link from 2-hybrid
    • natural language analysis
  • directed links
    • classical genetic gene-pathway analysis can add direction to some links
    • Chen, et al. (2004) network links inferred by dynamic correlation of expression level between txn factor and other gene
    • chip-chip (chromatin IP, hybridize labeled frags to array), direction is a priori based on txn factor affects gene expression
• Lee, et al., Science 2002, Whitehead Institute.
  • mapped targets of 106 txn factors using chromatin immunoprecipitation
  • Would be hard using this model; would have to supplement txn-factor-to-upstream-region interaction map with other information, e.g., go terms, microarray data
• Climescu and Quirk, 2007, Grenoble and Los Alamos
  • uses data from Chen, et al. at http://www.csie.ntu.edu.tw/~b89x035/yeast/
• http://www.ncbi.nlm.nih.gov/sites/entrez?Db=pubmed&Cmd=ShowDetailView&TermToSearch=15802287

Research teams to recruit

What are the research teams available to address this challenge?

Team leader	email	affiliation	website	publication	Status
K.V. Venkatesh	venks@iitb.ac.in	IIT Bombay	Venkatesh group	Rawool and Venkatesh, 2007	NA
Richard A. Young	young@wi.mit.edu	Whitehead Institute	NA	Lee, et al., 2002	NA
Timothy R. Hughes	t.hughes@utoronto	U Toronto	Hughes Lab	NA	NA
Gad Shaulsky	gadi@bcm.tmc.edu	UT Houston	lab home	NA	NA
Ju Han Kim	juhan@snu.ac.kr	Seoul	NA	Ohn, et al., 2007	NA
Bor-Sen Chen	bschen@moti.ee.nthu.edu.tw	Taiwan	data (excel) and software (matlab)	Chen, et al., 2004	NA
Alvis Brazma	brazma@ebi.ac.uk	EMBL-EBI, Cambridge, UK	NA	Schlitt and Brazma, 2007	NA
Adrianna Climescu-Haulica	adriana.climescu@cea.fr	Grenoble, France	NA	Climescu-Haulica and Quirk, 2007	NA
Carsten Peterson	carsten@thep.lu.se	Lund, Sweden	NA	Kauffman, et al., 2003	NA
Gustavo Stolovitzky	gustavo@us.ibm.com	IBM	homepageDREAM wiki	NA	NA
Andrea Califano	califano@c2b2.columbia.edu	Columbia U	labDREAM wiki	NA	NA
NA	NA	NA	NA	NA	NA

Other partners to consider

• Statistics for microarrays, e.g., John Lu
• legal issues (see CASP notes)
• web server design and implementation
• DREAM organization (evaluation challenges), could provide feedback, reach target audience

Recruitment of partners

• initial emails providing easy-to-read overview
• possible discussion on DREAM discussion board
• phone calls if that doesn't work; follow up

Feedback and additional thoughts on Yeast knockout project plan

Analysis of Hughes, 2000 results?

The Hughes paper from Rosetta Inpharmatics presents results

• 276 knockouts, not chosen at random
• 1 growth condition (complete plus glucose)
• table 1 has summary statistics on growth phenotypes and numbers of genes significantly affected
• supplementary data available from Rosetta web site: http://www.rii.com/publications/2000/cell_hughes.html

The results show that, even for mutants that do not have a noticeable growth phenotype, and even using a gene-specific error model, 29 % of mutant profiles are affected at 5 or more genes.

DREAM discussion board

Started discussion 11 March, didn't get feedback for first two weeks. Two comments in April.

Discussion with John Moult, CASP organizer

1. experimental design
   • need to know distribution of effects-- how many knockouts will have substantial effect?
     • one way to estimate: look at old knockout expression profiles from ref
     • another way: look at fitness effects from competitive growth experiments
2. publicity
   • contact DREAM for access to mailing-list participants
   • make arrangement in advance with journal to publish results (try to get pre-contest publication, too)
   • publish again in a year
3. infrastructure for prediction phase
   • issue receipt, number for id purposes (don't let participants say afterwards "I uploaded a prediction and you lost it")
4. scoring
   • CASP allows 5 predictions per target, allows group to take best score out of 5.
   • organizers discuss alternative scoring schemes (participants joke that for every team, there is a score function that makes it the winner)
5. workshop
   • don't release results until workshop (for dramatic effect)

Additional thoughts

* How do we design the experiment to maximize benefits to the community?

• not too easy, not too hard
• maximize expected Var(success_by_team), i.e., maximize distinction between different strategies
• requires systems-level model to achieve success (hard to game the system)

* Logistics and management of predictions and analysis

• • server sends received prediction back to submitter with time-limited request for corrections (else prediction is as received)
  • solicit alternative scoring schemes in advance
  • workshop is crucial-- need to get estimate of costs

Major project

CAMP Web Utilities

•   - advanced search
• WebTopicList - all topics in alphabetical order
• WebChanges - recent topic changes in this web
• WebNotify - subscribe to an e-mail alert sent when topics change
• WebRss, WebAtom - RSS and ATOM news feeds of topic changes
• WebStatistics - listing popular topics and top contributors
• WebPreferences - preferences of this web