Are we overfitting to our validation data?  How can we do better?

This one-day knowledge-exchange workshop to explore this question, primarily in a medical context, drawing together people from different perspectives (statistics, pharma and machine learning practitioners) to collaborate on this methodological issue and to develop more effective ways to use our medical (or other) data.  The ambition of this day is to build the network needed to write a consensus paper on this topic, targeted at a broad-reach journal.  Once the paper has been accepted, the intention is to run a follow-on symposium on the subject for a broader range of participants to discuss and disseminate our results, thereby helping to improve the practice of building ML models in medicine.

This is an invitation-only event.  if you would like to contribute, please contact Julian Gilbey at jdg18@cam.ac.uk

Draft schedule

9:30-10:00am    Arrival, registration, refreshments

10:00am            Welcome, scene setting and intended outputs of the day

10:15am            Clinical trials (*)

11:15am            Statistics and Machine Learning development: p-values, AUC,

12:30pm            Lunch

1:30pm              Theoretical bounding of errors for ML models, for example PAC

2:45pm              Paper planning

3:00pm              Tea & coffee break

3:30pm              Regulatory aspects of model training (*)

4:00pm              Publicity, translation, impact beyond; next steps

4:15pm             “Unconference”: 2 min talks, what have we missed, …

4:55pm             Wrap-up

5:00pm             Drinks reception

6:00pm             Dinner at Stazione.