The Working Group is investigating developing a partnership with MAQC (www.maqcsociety.org). More details here:
- Title: An Internationally-conducted External Quality Control scheme on Machine Learning Algorithms to assess Tumor Infiltrating Lymphocytes in Breast Cancer.
- The project coordinator (name, affiliation, and email address): Roberto Salgado (lead), GZA Department of Pathology, Belgium, roberto@salgado.be, Peter Mac Callum Cancer Center, Melbourne, Australia; BCTL, Jules Bordet Institute, Belgium; Peter Savas, Peter Mac Callum Cancer Center, Melbourne, Australia; and peter.savas@petermac.org; Frederick Klaushen, Charité Berlin,Frederick.Klauschen@charite.de, Germany; Wentao Yang, yangwt2000@163.com, Fudan University, China; Stefan Michiels, Stefan.MICHIELS@gustaveroussy.fr, Paris, France; Scooter Willis, willishf@gmail.com, Avera Cancer Institute, David Rimm, david.rimm@yale.edu, Yale University/US, Benjamin Haibe-Kans, benjamin.haibe.kains@utoronto.ca, University of Toronto, Canada; Cesare Furlanello, furlan@fbk.eu, MBPA, Tina Gruosso, tina.gruosso@mail.mcgill.ca, Goodman Cancer Research Centre, McGill University, Canada.
- Instruction on how to participate in the project (and award): all groups with documented analytically validated machine learning tools are welcome to participate. Interested groups need to provide to the coordinators a motivated request to participate, with documentation of the analytical validity of the method they are going to apply for this program. The information provided by the groups will be considered strictly confidential. The method that is going to applied for this assessment needs to be locked, may not be changed during the assessment and should be described in detail in order to avoid implicit overfitting.
- Objectives/Goals:
- To set quality standards and performance metrics on machine learning algorithms before introduction in a clinical trial setting and/or daily practice setting.
- To set quality standards and performance metrics that can be used by regulatory agencies to certify machine learning algorithms for use in patient management.
- To develop a framework for comparison of machine learning algorithms to determine precise quantitative metrics of other breast cancer biomarkers, like Ki67.
- Background:
At present, in early-stage disease clinico-pathological risk stratification is performed using a limited set of features such as tumor size and lymph node status. Very large adjuvant trials such as ALTTO and APHINITY that have applied these stratification schemes have illustrated the key problems with the current classification scheme – it does not stratify patients with sufficient granularity to permit selection for clinical trials. The current scheme also takes the approach of placing patients on a continuum of risk. This is at odds with results from high-throughput technologies such as gene expression profiling and genomic assays, which focus on identifying individual patient groups with particular clinical behaviour. Several results in this area have identified genomic, transcriptomic or proteomic features which in hindsight are associated with particular histological features. This suggests that the histological appearance of a tumor represents a useful cancer phenotype which can be further explored, and contribute to staging and stratification.
Machine learning refers to the general computational approach whereby data is used by algorithms to develop predictive models. These models are finely tuned to optimize accuracy and generalizability as applied to new data. Although machine learning existed for some time, more recently, advances in algorithm development and hardware infrastructure has enabled ‘deep learning’ approaches. Deep learning was originally designed to mimic the neural architecture of the human brain, and conceptually uses a series of connected nodes (neural nets) which respond to input in a way that is tuned with repeated cycles of learning. Neural nets have the ability to learn rich representations of complex data, which may contain hierarchical and non-linear relationships. These abilities make neural nets ideally suited to image classification. They have exhibited spectacular results in this area, often matching the performance of experts in the field or exceeding it (superhuman capabilities).
With recent advances in deep learning provides a path forward for numerous applications in digital pathology. On one level, the robust performance and training characteristics of deep learning allows us to develop accurate automated assays for pathological features such as grade and lymphocyte infiltration. These have the potential to be ‘learn once, apply everywhere’. This is in contrast to existing imaging methods, which lack the precision and robustness to be used in the clinical setting. If the promise of deep learning can be validated, the use of digital pathology would aid pathologists in routine reporting, and could be expected to improve the validity of current pathology based clinico-pathological features. In the short term, digital pathology would also help standardize pathology results within and across trials given the time required for pathology assessed quantitative metrics.
TILs have been shown to be a reliable and reproducible marker of tumor immunogenicity in breast cancer. It is clear that higher levels of TILs are associated with improved prognosis in early stage TNBC and HER2-positive breast cancer, as well as a higher probability of achieving pCR in the neoadjuvant setting. Analysis of TILs in residual disease specimens after neoadjuvant therapy has also been shown to have prognostic value. The evaluation of TILs as a biomarker in breast cancer is likely to be extended from the research domain to the clinical setting in the near future. The assessment of TILs by digital image analysis might be useful for standardization in the future, since this approach has the potential, for example, to determine the number of TILs per mm² stromal tissue as an exact measurement contrary to the approximate semi-quantitative evaluation suggested at this moment. In the first International Guidelines on TIL-assessment in breast cancer we proposed to develop an inter-laboratory Ring study to assess the reproducibility and clinical validity of TILs assessment, including machine learning algorithms. While TILs have been measured morphologically and have been shown to add predominantly prognostic information, methodological open questions in the morphological evaluation of TILs still remain, for example the assessment and importance of spatial TIL-heterogeneity. The measurement on H&E-stained slides most likely represents the beginning of the efforts to use infiltrating cell properties as companion diagnostic tests. Thus, as a field, we should be open to the introduction of molecular methods, most likely in situ, that can classify the TILs-component and bring higher levels of information to the patient sample. However, at this time, these deep learning approaches are still experimental and not sufficiently documented for introduction into standard practice.
On another level however, deep learning also permits discovery of image based features which may be very difficult for current approaches to identify, particularly if they only exist in small groups of patients. The key benefit of deep learning here is to rapidly identify pathological features in clinical trials that are predictive of treatment or prognostic of outcome in a standardized way. This is an essential first step in deciding if previously undescribed pathological features are clinically relevant, and is largely infeasible using current approaches. Deep learning also permits modification and retraining of the feature set to optimize accuracy and interpretability, which is again infeasible with current methods.
The Working Group is therefore proposing a collaboration with the Massive Analysis and Quality Control Consortium (www.maqcsociety.org) characterizing tumor infiltrating lymphocytes using machine learning algorithms. Developing a machine learning based assay for tumor infiltrating lymphocytes would enable rapid expansion of this promising pathological feature, and by providing an adjunct to human pathologists, enhance the validity and robustness for prognosis/prediction.
- Specific Aims:
- Comparison of the machine learning image classification metrics with those of pathologists in the RING-study which the Working group has published (Carsten Denkert et al., Mod. Pathol. 2016)
- Comparison of automated TILs scoring with pathologist scoring results in different settings, namely core biopsies, full sections, pre-invasive (DCIS), untreated and treated tumors.
- Comparing the performance of deep learning approaches to identify complex features such as clustering/spatial statistics including proximity of TILs to cancer cells that are prognostic of outcome or predictive of treatment.
- Comparing the clinical validity of different machine learning algorithms, the utility of combining models for improved accuracy and to identify possible false positives and false negatives.
- To combine annotated training data from different sites to create a comprehensive breast cancer ML training and validation data base hosted by the consortium.
- A framework will be developed to facilitate automated testing, validation and certification of image classification derived pathology metrics that can improve the standard of care.
- Develop together with both groups a kind of review, perspective or opinion paper on the use of Artificial Intelligence/machine learning tools in Oncology, focusing but not exclusively on TILs and including the quality requirements for use of these technologies in a clinical trial and daily practice setting, similar in kind as the Lisa Mc Shane paper in Nature Criteria for the use of omics-based predictors in clinical trials, doi: 10.1038/nature12564, Nature 2013, which is an exercise that may be very useful for regulatory (FDA; EMA).
- Study Design, including a description of the dataset, endpoints, general methodologies, how to report the results and in which reporting format.
- Breast cancer slide-sets in different settings (invasive, DCIS, residual disease) with known TIL-assessment by pathologists will be posted on the website of the International Immune-oncology Biomarker Group.
- A clinical trial slide-set, with clinical annotation, will be hosted on the website of the International Immune-oncology Biomarker Group.
- All these slides can then be assessed by all participating groups.
- The metrics assessed will be reported on pre-specified formats to the coordinators.
- A systematic comparison of the output of the machine learning/deep learning approaches with the pathologists’ score will be performed in all datasets and eventual added clinical validity to the pathologists’ TIL-score will be evaluated using the clinical trial datasets.
- Timeline
- The project is aimed to start in 2019 and aims to be finished within 1 year from the start of the program.
- Results will be presented at the annual meeting of the International Immuno-Oncology Biomarker Working Group held at the San Antonio Breast Cancer Conference and at the annual MAQC-Conference.
- Publication is aimed within 6 months after completion of the program in a high level journal.
online cialis https://graph.org/Omicron-Variant-Symptoms-Is-An-Excessive-Amount-Of-Mucus-A-COVID-19-Symptom-02-24
Whoa a good deal of helpful information!
Generic cialis tadalafil https://kertvbs.webgarden.com/
You actually explained it effectively.
cialis 5mg prix https://andere.strikingly.com/
With thanks! I enjoy this!
canada rx https://gerweds.over-blog.com/2022/03/modeling-covid-19-mortality-risk-in-toronto-canada-sciencedirect.html
Thanks a lot. I enjoy this.
northwest pharmacy canada https://keuybc.estranky.sk/clanky/30-facts-you-must-know–a-covid-cribsheet.html
Reliable write ups. Thanks a lot.
cialis pills https://gwertvb.mystrikingly.com/
Wow plenty of fantastic facts!
Cialis tadalafil https://telegra.ph/Is-It-Safe-To-Lift-COVID-19-Travel-Bans-04-06
Thanks a lot. Plenty of forum posts!
cialis online https://graph.org/The-Way-To-Get-Health-Care-At-Home-During-COVID-19—Health–Fitness-04-07
Amazing loads of superb advice!
canadian pharmacies online prescriptions https://chubo3.wixsite.com/canadian-pharmacy/post/what-parents-must-find-out-about-kids-and-covid-19
Cheers! I value this.
canadian online pharmacies https://canadian-pharmacies0.yolasite.com/
Kudos, Numerous postings.
cialis 20 mg https://pharmacy-online.yolasite.com/
Good facts. Thanks!
tadalafil https://kevasw.webgarden.com/
You expressed it terrifically.
northwest pharmacy canada https://62553dced4718.site123.me/
Awesome stuff. Kudos!
buy cialis without a doctor’s prescription https://seketu.gonevis.com/high-10-tips-with-order-medicine-online-1/
You revealed this effectively!
canadian cialis https://site128620615.fo.team/
Helpful tips. Appreciate it!
canadian drugs https://625a9a98d5fa7.site123.me/blog/age-dependence-of-healthcare-interventions-for-covid-19-in-ontario-canada
Fantastic information. Thanks a lot.
trust pharmacy canada https://hernswe.gonevis.com/scientists-model-true-prevalence-of-covid-19-throughout-pandemic/
Thanks, Wonderful stuff!
no 1 canadian pharcharmy online https://selomns.gonevis.com/a-modified-age-structured-sir-model-for-covid-19-type-viruses/
Many thanks! Lots of tips.
tadalafil generic https://fwervs.gumroad.com/
You actually stated that exceptionally well!
purchasing cialis on the internet https://trosorin.mystrikingly.com/
Nicely put. Cheers.
buy cialis online https://sasnd0.wixsite.com/cialis/post/impotent-victims-can-now-cheer-up-try-generic-tadalafil-men-health
Kudos, Numerous info.
canadian cialis https://626106aa4da69.site123.me/blog/new-step-by-step-roadmap-for-tadalafil-5mg
Truly a good deal of valuable information!
tadalafil generic https://generic-cialis-20-mg.yolasite.com/
Good material. Appreciate it!
Cialis tadalafil https://hsoybn.estranky.sk/clanky/tadalafil-from-india-vs-brand-cialis—sexual-health.html
You stated it well.
tadalafil tablets https://skuvsbs.gonevis.com/when-tadalafil-5mg-competitors-is-good/
Seriously a lot of terrific info.
tadalafil https://hemuyrt.livejournal.com/325.html
You reported that adequately!
cialis 5mg prix https://site373681070.fo.team/
Thanks a lot! An abundance of postings.
buy cialis online https://sehytv.wordpress.com/
Nicely put. Many thanks.
prescription drugs without prior prescription https://ghswed.wordpress.com/2022/04/27/he-final-word-information-to-online-pharmacies/
You actually expressed it fantastically.
canadian pharmacies shipping to usa https://kerbgsw.mystrikingly.com/
You suggested this superbly.
cialis generico online https://kuebser.estranky.sk/clanky/supereasy-methods-to-study-every-part-about-online-medicine-order-discount.html
You actually suggested this superbly!
buy cialis without a doctor’s prescription https://kewertyn.wordpress.com/2022/04/27/expect-more-virtual-house-calls-out-of-your-doctor-thanks-to-telehealth-revolution/
Nicely put. Appreciate it!
cialis prices https://kerbiss.wordpress.com/2022/04/27/14/
Nicely voiced without a doubt! .
tadalafil 5mg https://heswcxc.wordpress.com/2022/04/30/online-medicine-tablets-shopping-promotion-one-hundred-and-one/
You actually mentioned that wonderfully.
cialis uk https://sernert.estranky.sk/clanky/confidential-information-on-online-pharmacies.html
Thank you, I enjoy it!
online cialis https://kertubs.mystrikingly.com/
You definitely made the point!
tadalafil generic https://626f977eb31c9.site123.me/blog/how-google-is-changing-how-we-approach-online-order-medicine-1
Cheers! I enjoy this.
canadian pharcharmy online https://canadian-pharmaceuticals-online.yolasite.com/
Amazing tons of valuable knowledge.
tadalafil https://online-pharmacies0.yolasite.com/
You said it perfectly..
cialis generic https://6270e49a4db60.site123.me/blog/the-untold-secret-to-mastering-aspirin-in-just-7-days-1
Really quite a lot of great data.
cialis generic https://deiun.flazio.com/
Fantastic information, Regards!
Cialis tadalafil https://kertyun.flazio.com/
Wonderful content. With thanks!
cialis lowest price https://kerntyas.gonevis.com/the-mafia-guide-to-online-pharmacies/
You actually stated it terrifically.
canadian pharmacies that are legit https://kerbnt.flazio.com/
Cheers. I appreciate it.
cialis uk http://nanos.jp/jmp?url=http://cialisonlinei.com/
Great posts. Thanks a lot!
Generic cialis tadalafil http://ime.nu/cialisonlinei.com
You suggested it adequately!
cialis pills https://womed7.wixsite.com/pharmacy-online/post/new-ideas-into-canada-pharmacies-never-before-revealed
Info nicely regarded!.
cialis 20 mg best price https://kerntyast.flazio.com/
Terrific info, Regards!
cialis lowest price https://sekyuna.gonevis.com/three-step-guidelines-for-online-pharmacies/
Kudos, Lots of facts!
cialis prices https://gewrt.usluga.me/
Truly loads of amazing information!
purchasing cialis on the internet https://pharmacy-online.webflow.io/
Fantastic facts. With thanks.
cialis 20mg prix en pharmacie https://canadian-pharmacy.webflow.io/
You revealed that terrifically.
canadian pharmacy viagra brand https://site273035107.fo.team/
Regards. Awesome stuff.
cialis uk https://site656670376.fo.team/
With thanks. Plenty of advice!
canadian pharmacy cialis https://site561571227.fo.team/
Nicely put, Kudos!
cialis generico online https://site102906154.fo.team/
Many thanks! Lots of content!
canadian pharmacy meds https://hekluy.ucraft.site/
Very good facts. Appreciate it.
generic for cialis https://kawsear.fwscheckout.com/
Awesome postings, Appreciate it.
cialis 20mg https://hertnsd.nethouse.ru/
You actually expressed it wonderfully.
generic for cialis https://uertbx.livejournal.com/402.html
Whoa lots of superb material.
buy generic cialis https://lwerts.livejournal.com/276.html
Wow all kinds of awesome facts!
canadian pharmacy viagra https://avuiom.sellfy.store/
Amazing quite a lot of fantastic tips.
canadian viagra https://pharmacies.bigcartel.com/
Awesome write ups. Thanks!
cialis pills https://kwersd.mystrikingly.com/
Thank you! I appreciate this.
Canadian Pharmacy USA https://gewsd.estranky.sk/clanky/drugstore-online.html
Thanks. Plenty of forum posts!
online canadian pharmacies https://kqwsh.wordpress.com/2022/05/16/what-everybody-else-does-when-it-comes-to-online-pharmacies/
Lovely facts, Many thanks!
medication without a doctors prescription http://site592154748.fo.team/
Information very well utilized!.
Cheap cialis https://lasert.gonevis.com/recommended-canadian-pharmacies-2/
Good facts, Thanks.
buy cialis without a doctor’s prescription http://aonubs.website2.me/
Incredible loads of excellent information.
canadadrugs https://dkyubn.bizwebs.com/
This is nicely said. .
buy cialis online https://asebg.bigcartel.com/canadian-pharmacy
Nicely put, Thanks a lot.
cialis from canada https://medicine-online.estranky.sk/clanky/understand-covid-19-and-know-the-tricks-to-avoid-it-from-spreading—–medical-services.html
Cheers. Numerous tips.
tadalafil 20mg https://disvaiza.mystrikingly.com/
You expressed it terrifically.
cialis 20 mg best price https://swenqw.company.site/
This is nicely said! .
buy viagra usa https://kewburet.wordpress.com/2022/04/27/how-to-keep-your-workers-healthy-during-covid-19-health-regulations/
Amazing loads of great facts.
cialis pills https://kaswesa.nethouse.ru/
Good facts. Cheers.
buy cialis online https://628f789e5ce03.site123.me/blog/what-everybody-else-does-when-it-comes-to-canadian-pharmacies
Seriously plenty of excellent knowledge!
cialis from canada https://canadian-pharmaceutical.webflow.io/
Regards! Lots of data.
Cialis tadalafil http://pamelaliggins.website2.me/
Lovely data, Thank you.
cialis tablets http://pharmacy.prodact.site/
Lovely postings. Thanks!
online prescriptions without a doctor https://hub.docker.com/r/gserv/pharmacies
Many thanks! Valuable information!
tadalafil generic https://hertb.mystrikingly.com/
Kudos, Loads of write ups!
online prescriptions without a doctor https://kedmnx.estranky.sk/clanky/online-medicine-tablets-shopping-the-best-manner.html
This is nicely said. !
tadalafil generic https://selera.mystrikingly.com/
Regards! I enjoy this.
canada pharmacy https://ksorvb.estranky.sk/clanky/why-online-pharmacies-is-good-friend-to-small-business.html
Very well voiced really. .
canadian online pharmacy https://gevcaf.estranky.cz/clanky/safe-canadian-online-pharmacies.html
Great material. Thank you!
Cheap cialis https://kwersv.proweb.cz/
Cheers! I like this!
tadalafil without a doctor’s prescription https://kwervb.estranky.cz/clanky/canadian-government-approved-pharmacies.html
You reported that superbly.
cialis prices https://sdtyli.zombeek.cz
Valuable stuff. Many thanks.
canadian prescription drugstore https://kwsde.zombeek.cz/
Beneficial advice. Kudos.
online cialis https://heklrs.wordpress.com/2022/06/14/canadian-government-approved-pharmacies/
You suggested that effectively.
online pharmacies https://iercvsw.wordpress.com/2022/06/14/canadian-pharmacies-the-fitting-manner/
This is nicely expressed. !
canadian pharmacies online https://site955305180.fo.team/
Thanks. I value this!
tadalafil 5mg http://site841934642.fo.team/
You have made your position quite well..
tadalafil 20 mg https://62b2f636ecec4.site123.me/blog/canadian-pharmaceuticals-online
Seriously tons of valuable facts!
cialis tablets https://62b2ffff12831.site123.me/blog/canadian-pharmaceuticals-for-usa-sales
Cheers, I enjoy this.