research-article

Open access

A Human-Centered Evaluation of a Deep Learning System Deployed in Clinics for the Detection of Diabetic Retinopathy

Authors:

Elizabeth Baylor,

Anna Iurchenko,

Paisan Ruamviboonsuk,

Laura M. VardoulakisAuthors Info & Claims

CHI '20: Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems

Pages 1 - 12

https://doi.org/10.1145/3313831.3376718

Published: 23 April 2020 Publication History

All formats PDF

Abstract

Deep learning algorithms promise to improve clinician workflows and patient outcomes. However, these gains have yet to be fully demonstrated in real world clinical settings. In this paper, we describe a human-centered study of a deep learning system used in clinics for the detection of diabetic eye disease. From interviews and observation across eleven clinics in Thailand, we characterize current eye-screening workflows, user expectations for an AI-assisted screening process, and post-deployment experiences. Our findings indicate that several socio-environmental factors impact model performance, nursing workflows, and the patient experience. We draw on these findings to reflect on the value of conducting human-centered evaluative research alongside prospective evaluations of model accuracy.

Supplementary Material

MP4 File (a589-beede-presentation.mp4)

Download
40.77 MB

References

[1]

Michael D. Abràmoff, Philip T. Lavin, Michele Birch, Nilay Shah, and James C Folk. 2018. Pivotal trial of an autonomous AI-based diagnostic system for detection of diabetic retinopathy in primary care offices. NPJ Digit Med 1 (Aug. 2018), 39.

[2]

E. Alberdi, A. A. Povyakalo, L. Strigini, P. Ayton, M. Hartswood, R. Procter, and R. Slack. 2005. Use of computer-aided detection (CAD) tools in screening mammography: a multidisciplinary investigation. Br. J. Radiol. 78 Spec No 1 (2005), S31--40.

[3]

American Academy of Ophthalmology. 2015. Eye Health Statistics. https://www.aao.org/newsroom/eye-health-statistics#_edn25. (2015). Accessed: 2019--9--7.

[4]

Eta S. Berner. 2007. Clinical Decision Support Systems: Theory and Practice. Springer Science & Business Media.

[5]

Hugh Beyer and Karen Holtzblatt. 1997. Contextual design: defining customer-centered systems. Elsevier.

Digital Library

[6]

Timothy W. Bickmore, Laura M. Pfeifer, and Brian W. Jack. 2009. Taking the Time to Care: Empowering Low Health Literacy Hospital Patients with Virtual Nurse Agents. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '09). ACM, NY, NY, USA, 1265--1274.

Digital Library

[7]

Keld Bødker, Finn Kensing, and Jesper Simonsen. 2009. Participatory IT design: designing for business and workplace realities. MIT press.

Digital Library

[8]

Carrie J. Cai, Emily Reif, Narayan Hegde, Jason Hipp, Been Kim, Daniel Smilkov, Martin Wattenberg, Fernanda Viegas, Greg S. Corrado, Martin C. Stumpe, and Michael Terry. 2019a. Human-Centered Tools for Coping with Imperfect Algorithms During Medical Decision-Making. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (CHI '19). ACM, NY, NY, USA, Article 4, 14 pages.

Digital Library

[9]

Carrie Jun Cai, Samantha Winter, David Steiner, Lauren Wilcox, and Michael Terry. 2019b. "Hello AI": Uncovering the Onboarding Needs of Medical Practitioners for Human-AI Collaborative Decision-Making. CSCW Conf Comput Support Coop Work 2019 (2019).

Digital Library

[10]

Ronald A Castellino. 2005. Computer aided detection (CAD): an overview. Cancer Imaging 5, 1 (2005), 17.

[11]

CDC. 2019. More than 100 million Americans have diabetes or prediabetes. Press Release. (February 2019). https://www.cdc.gov/media/releases/2017/ p0718-diabetes-report.html.

[12]

Elodia B. Cole, Zheng Zhang, Helga S. Marques, R. Edward Hendrick, Martin J. Yaffe, and Etta D. Pisano. 2014. Impact of Computer-Aided Detection Systems on Radiologist Accuracy With Digital Mammography. AJR Am. J. Roentgenol. 203, 4 (Oct. 2014), 909.

[13]

Kathleen Musante DeWalt and Billie R DeWalt. 2002. Participant Observation: A Guide for Fieldworkers. Rowman Altamira.

[14]

Shelley E Ellis, Theodore Speroff, Robert S. Dittus, Anne Brown, James W. Pichert, and Tom A. Elasy. 2004. Diabetes patient education: a meta-analysis and meta-regression. Patient Educ. Couns. 52, 1 (Jan. 2004), 97--105.

[15]

Glyn Elwyn, Isabelle Scholl, Caroline Tietbohl, Mala Mann, Adrian GK Edwards, Catharine Clay, France Légaré, Trudy van der Weijden, Carmen L. Lewis, Richard M. Wexler, and others. 2013. "Many miles to go...": a systematic review of the implementation of patient decision support interventions into routine clinical practice. BMC medical informatics and decision making 13, 2 (Nov. 2013), 1--10.

[16]

Geraldine Fitzpatrick and Gunnar Ellingsen. 2013. A review of 25 years of CSCW research in healthcare: contributions, challenges and future agendas. Computer Supported Cooperative Work (CSCW) 22, 4--6 (2013), 609--665.

Digital Library

[17]

Jodi Forlizzi and John Zimmerman. Promoting service design as a core practice in interaction design.

[18]

Maryellen L Giger, Heang-Ping Chan, and John Boone. 2008. Anniversary Paper: History and status of CAD and quantitative image analysis: The role of Medical Physics and AAPM. Med. Phys. 35, 12 (Dec. 2008), 5799.

[19]

Trisha Greenhalgh, Joe Wherton, Chrysanthi Papoutsi, Jenni Lynch, Gemma Hughes, Sue Hinder, Rob Procter, Sara Shaw, and others. 2018. Analysing the role of complexity in explaining the fortunes of technology programmes: empirical application of the NASSS framework. BMC medicine 16, 1 (2018), 66.

[20]

Varun Gulshan, Lily Peng, Marc Coram, Martin C Stumpe, Derek Wu, Arunachalam Narayanaswamy, Subhashini Venugopalan, Kasumi Widner, Tom Madams, Jorge Cuadros, and others. 2016. Development and validation of a deep learning algorithm for detection of diabetic retinopathy in retinal fundus photographs. Jama 316, 22 (2016), 2402--2410.

[21]

Mark Hartswood, Rob Procter, Mark Rouncefield, Roger Slack, James Soutter, and Alex Voss. 2003. ?Repairing' the Machine: A Case Study of the Evaluation of Computer-Aided Detection Tools in Breast Screening. In ECSCW 2003. Springer, 375--394.

[22]

Matthew K Hong, Clayton Feustel, Meeshu Agnihotri, Max Silverman, Stephen F Simoneaux, and Lauren Wilcox. 2017. Supporting families in reviewing and communicating about radiology imaging studies. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems. ACM, 5245--5256.

Digital Library

[23]

Jenchitr W, Hanutsaha P, Iamsirithaworn S, Parnrat U, Choosri P. 2007. The national survey of blindness low vision and visual impairment in thailand 2006--2007. Thai J Pub Hlth Ophthalmol 21, 1 (2007), 10.

[24]

Marina Jirotka, Rob Procter, Mark Hartswood, Roger Slack, Andrew Simpson, Catelijne Coopmans, Chris Hinds, and Alex Voss. 2005. Collaboration and trust in healthcare innovation: The eDiaMoND case study. Computer Supported Cooperative Work (CSCW) 14, 4 (2005), 369--398.

Digital Library

[25]

Pearse A Keane and Eric J Topol. 2018. With an eye to AI and autonomous diagnosis. npj Digital Medicine 1, 1 (Aug. 2018), 1--3.

[26]

Ajay Kohli and Saurabh Jha. 2018. Why CAD failed in mammography. Journal of the American College of Radiology 15, 3 (2018), 535--537.

[27]

Mark A Musen, Blackford Middleton, and Robert A Greenes. 2014. Clinical Decision-Support Systems. In Biomedical Informatics. Springer, London, 643--674.

[28]

American Academy of Ophthalmology. 2002. International Clinical Diabetic Retinopathy Disease Severity Scale. http://www.icoph.org/dynamic/attachments/resources/ diabetic-retinopathy-detail.pdf. (Oct. 2002). Accessed: 2019--12--17.

[29]

World Health Organization. 2018. Vision impairment and blindness. https://www.who.int/news-room/ fact-sheets/detail/blindness-and-visual-impairment. (2018). Accessed: 2019--9--13.

[30]

Sun Young Park, Pei-Yi Kuo, Andrea Barbarin, Elizabeth Kaziunas, Astrid Chow, Karandeep Singh, Lauren Wilcox, and Walter Lasecki. 2019. Identifying Challenges and Opportunities in Human--AI Collaboration in Healthcare. (2019).

[31]

Laura Pfeifer Vardoulakis, Amy Karlson, Dan Morris, Greg Smith, Justin Gatewood, and Desney Tan. 2012. Using mobile phones to present medical information to hospital patients. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. ACM, 1411--1420.

Digital Library

[32]

Sarah K Pontefract, Jamie J Coleman, Hannah K Vallance, Christine A Hirsch, Sonal Shah, John F Marriott, and Sabi Redwood. 2018. The impact of computerised physician order entry and clinical decision support on pharmacist-physician communication in the hospital setting: A qualitative study. PloS one 13, 11 (2018), e0207450.

[33]

Paisan Raumviboonsuk, Jonathan Krause, Peranut Chotcomwongse, Rory Sayres, Rajiv Raman, Kasumi Widner, Bilson JL Campana, Sonia Phene, Kornwipa Hemarat, Mongkol Tadarati, and others. 2019. Deep learning versus human graders for classifying diabetic retinopathy severity in a nationwide screening program. npj Digital Medicine 2, 1 (2019), 25.

[34]

Madhu C Reddy, David W McDonald, Wanda Pratt, and M Michael Shabot. 2005. Technology, work, and information flows: Lessons from the implementation of a wireless alert pager system. Journal of biomedical informatics 38, 3 (2005), 229--238.

Digital Library

[35]

Nigam H. Shah, Arnold Milstein, and Steven C. Bagley, PhD. 2019. Making Machine Learning Models Clinically Useful. JAMA (08 2019).

[36]

Lauren Wilcox, Dan Morris, Desney Tan, and Justin Gatewood. 2010. Designing patient-centric information displays for hospitals. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. ACM, 2123--2132.

Digital Library

[37]

Lauren Wilcox, Janet Woollen, Jennifer Prey, Susan Restaino, Suzanne Bakken, Steven Feiner, Alexander Sackeim, and David K Vawdrey. 2016. Interactive tools for inpatient medication tracking: a multi-phase study with cardiothoracic surgery patients. J. Am. Med. Inform. Assoc. 23, 1 (Jan. 2016), 144--158.

[38]

World Health Organization. 2007. Global Initiative for the Elimination of Avoidable Blindness : action plan 2006--2011. https://www.who.int/blindness/Vision2020_report.pdf. (2007). Accessed: 2019--9--7.

[39]

World Health Organization. 2014. WHO: Diabetes factsheet. https://www.who.int/news-room/fact-sheets/detail/diabetes. (2014). Accessed: 2019--9--13.

[40]

World Health Organization. 2016a. Diabetes country profiles 2016 : Thailand. https://www.who.int/diabetes/ country-profiles/tha_en.pdf?ua=1. (2016). Accessed: 2019--9--7.

[41]

World Health Organization. 2016b. Diabetes country profiles 2016 : USA. https://www.who.int/diabetes/country-profiles/usa_en.pdf. (2016). Accessed: 2019--9--7.

[42]

Qian Yang, Aaron Steinfeld, and John Zimmerman. 2019. Unremarkable AI: Fitting Intelligent Decision Support into Critical, Clinical Decision-Making Processes. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (CHI '19). ACM, NY, NY, USA, Article 238, 11 pages.

Digital Library

Cited By

Thakur N(2024)Evolution of Generative AI in HealthcareRevolutionizing the Healthcare Sector with AI10.4018/979-8-3693-3731-8.ch002(26-53)Online publication date: 14-Jun-2024
https://doi.org/10.4018/979-8-3693-3731-8.ch002
Shanthalakshmi Revathy JMangaiyarkkarasi J(2024)Healthcare 5.0Federated Learning and Privacy-Preserving in Healthcare AI10.4018/979-8-3693-1874-4.ch015(235-256)Online publication date: 19-Apr-2024
https://doi.org/10.4018/979-8-3693-1874-4.ch015
Onalaja JShahra EBasurra SJabbar W(2024)Image Classifier for an Online Footwear Marketplace to Distinguish between Counterfeit and Real Sneakers for ResaleSensors10.3390/s2410303024:10(3030)Online publication date: 10-May-2024
https://doi.org/10.3390/s24103030
Show More Cited By

Index Terms

A Human-Centered Evaluation of a Deep Learning System Deployed in Clinics for the Detection of Diabetic Retinopathy
1. Human-centered computing
  1. Human computer interaction (HCI)
    1. Empirical studies in HCI

Recommendations

Experiences in mHealth for chronic disease management in 4 countries
ISABEL '11: Proceedings of the 4th International Symposium on Applied Sciences in Biomedical and Communication Technologies

This paper describes mHealth applications to deal with Non Communicable Diseases in North and Latin America: In Chile, a project focused on Diabetes Mellitus type 2; In the United States, Honduras, and Mexico, projects focused in diabetes, heart failure,...
Diabetic Retinopathy Detection using Deep Learning
ICMLSC '20: Proceedings of the 4th International Conference on Machine Learning and Soft Computing

Diabetic Retinopathy (DR) is an eye disease associated with chronic diabetes. DR is the leading cause of blindness among working aged adults around the world and estimated it may affect more than 93 million people. Progression to vision impairment can ...
System Support of Diabetic Self-treatments Using Mobile Phones
Proceedings of the 2006 conference on Knowledge-Based Software Engineering: Proceedings of the Seventh Joint Conference on Knowledge-Based Software Engineering

We developed a system which helps diabetic people making self-treatments in their home, by exchanging treatment information and advices with their doctors and nurses in the hospital. To cure diabetes, it is essential to make daily treatment such as ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

CHI '20: Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems

April 2020

10688 pages

ISBN:9781450367080

DOI:10.1145/3313831

General Chairs:
Regina Bernhaupt
Eindhoven University of Technology, Netherlands
,
Florian 'Floyd' Mueller
Monash University, Australia
,
David Verweij
Newcastle University, UK
,
Josh Andres
RMIT, Australia
,
Program Chairs:
Joanna McGrenere
University of British Columbia, Canada
,
Andy Cockburn
University of Canterbury, New Zealand
,
Ignacio Avellino
University of Maryland Baltimore County, USA
,
Alix Goguey
Grenoble Alpes University, France
,
Pernille Bjørn
University of Copenhagen, Denmark
,
Shengdong (Shen) Zhao
National University of Singapore, Singapore
,
Briane Paul Samson
Future University Hakodate, Japan & De La Salle University, Philippines
,
Rafal Kocielnik
University of Washington, USA

Copyright © 2020 Owner/Author.

Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for third-party components of this work must be honored. For all other uses, contact the Owner/Author.

Sponsors

SIGCHI: ACM Special Interest Group on Computer-Human Interaction

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 23 April 2020

Check for updates

Badges

Honorable Mention

Author Tags

Qualifiers

Research-article

Conference

CHI '20

Sponsor:

SIGCHI

CHI '20: CHI Conference on Human Factors in Computing Systems

April 25 - 30, 2020

HI, Honolulu, USA

Acceptance Rates

Overall Acceptance Rate 6,199 of 26,314 submissions, 24%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

255
Total Citations
View Citations
20,244
Total Downloads

Downloads (Last 12 months)3,382
Downloads (Last 6 weeks)381

Reflects downloads up to 05 Aug 2024

Other Metrics

View Author Metrics

Citations

Cited By

Thakur N(2024)Evolution of Generative AI in HealthcareRevolutionizing the Healthcare Sector with AI10.4018/979-8-3693-3731-8.ch002(26-53)Online publication date: 14-Jun-2024
https://doi.org/10.4018/979-8-3693-3731-8.ch002
Shanthalakshmi Revathy JMangaiyarkkarasi J(2024)Healthcare 5.0Federated Learning and Privacy-Preserving in Healthcare AI10.4018/979-8-3693-1874-4.ch015(235-256)Online publication date: 19-Apr-2024
https://doi.org/10.4018/979-8-3693-1874-4.ch015
Onalaja JShahra EBasurra SJabbar W(2024)Image Classifier for an Online Footwear Marketplace to Distinguish between Counterfeit and Real Sneakers for ResaleSensors10.3390/s2410303024:10(3030)Online publication date: 10-May-2024
https://doi.org/10.3390/s24103030
Kawasaki R(2024)How Can Artificial Intelligence Be Implemented Effectively in Diabetic Retinopathy Screening in Japan?Medicina10.3390/medicina6002024360:2(243)Online publication date: 30-Jan-2024
https://doi.org/10.3390/medicina60020243
Bürger VAmann JBui CFehr JMadai V(2024)The unmet promise of trustworthy AI in healthcare: why we fail at clinical translationFrontiers in Digital Health10.3389/fdgth.2024.12796296Online publication date: 18-Apr-2024
https://doi.org/10.3389/fdgth.2024.1279629
Crabtree AMcGarry GUrquhart L(2024)Trust & AI? The Incalculable Calculus of RiskSSRN Electronic Journal10.2139/ssrn.4744387Online publication date: 2024
https://doi.org/10.2139/ssrn.4744387
Skevas Cde Olaguer NLleó AThiwa DSchroeter ULopes IMautone LLinke SSpitzer MYap DXiao D(2024)Implementing and evaluating a fully functional AI-enabled model for chronic eye disease screening in a real clinical environmentBMC Ophthalmology10.1186/s12886-024-03306-y24:1Online publication date: 1-Feb-2024
https://doi.org/10.1186/s12886-024-03306-y
Hussain WMabrok MGao HRabhi FRashed E(2024)Revolutionising healthcare with artificial intelligence: A bibliometric analysis of 40 years of progress in health systemsDIGITAL HEALTH10.1177/2055207624125875710Online publication date: 28-May-2024
https://doi.org/10.1177/20552076241258757
Gjødsbøl IRinggaard AHolm PBrunak SBundgaard H(2024)The robot butler: How and why should we study predictive algorithms and artificial intelligence (AI) in healthcare?DIGITAL HEALTH10.1177/2055207624124167410Online publication date: 24-Mar-2024
https://doi.org/10.1177/20552076241241674
Guo TLiu KZou HXu XYang JYu Q(2024)Refined image quality assessment for color fundus photography based on deep learningDIGITAL HEALTH10.1177/2055207623120758210Online publication date: 29-Feb-2024
https://doi.org/10.1177/20552076231207582
Show More Cited By

View Options

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

HTML Format

View this article in HTML Format.

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Media

Figures

Other

Tables

View Table of Contents