Technology & Operations Management
-
- 2024
- Working Paper
The Explainability Edge? A Field Experiment on Generative AI-Augmented Evaluations of Early-Stage Innovations
By: Jacqueline N. Lane, Léonard Boussioux, Charles Ayoubi, Ying Hao Chen, Camila Lin, Rebecca Spens, Pooja Wagh and Pei-Hsin WangThe rise of generative artificial intelligence (AI) has transformed creative problem-solving, necessitating a reassessment of idea evaluation processes to effectively screen and select from the resulting abundance of solutions. This study investigates how human-AI collaboration can enhance early-stage evaluations of innovative solutions, examining the interplay between objective criteria, which are quantifiable and measurable, and subjective criteria, which are based on personal opinion and intuition. We conducted a field experiment in partnership with MIT Solve, a marketplace for social impact innovation. The study involved 72 experts and 156 community screeners evaluating 48 real-world solutions for the 2024 Global Health Equity Challenge, resulting in 3,002 screener-solution pairs. We utilized GPT-4, a state-of-the-art large language model, to provide recommendations and explanations for screening decisions. Our experiment compared a human-only control condition against two AI-assisted treatments: a black box AI providing recommendations without explanations and an explainable AI offering both recommendations and rationales for its decisions. Our findings reveal that screeners strategically use AI insights, validating AI’s recommendations when they agree and scrutinizing AI recommendations when they disagree. Screeners assisted by AI were roughly 9 percentage points more likely to fail a solution than the control condition, primarily influenced by AI’s more stringent failure recommendations. Notably, when AI provided explanations for subjective criteria failures, screeners were 12 percentage points more likely to adhere to these recommendations compared to the black box treatment condition. Moreover, the reported effects were larger among the community than expert screeners. Data from interviews and mouse tracking reveal that AI explanations for subjective criteria led screeners to doubt their own human judgment and possibly over-rely on AI’s explanations. This research suggests a possible framework for human-AI collaboration in creative evaluation, where AI corroborates human judgment on objective criteria and humans retain primary responsibility for subjective assessments supported by AI insights.
- 2024
- Working Paper
The Explainability Edge? A Field Experiment on Generative AI-Augmented Evaluations of Early-Stage Innovations
By: Jacqueline N. Lane, Léonard Boussioux, Charles Ayoubi, Ying Hao Chen, Camila Lin, Rebecca Spens, Pooja Wagh and Pei-Hsin WangThe rise of generative artificial intelligence (AI) has transformed creative problem-solving, necessitating a reassessment of idea evaluation processes to effectively screen and select from the resulting abundance of solutions. This study investigates how human-AI collaboration can enhance early-stage evaluations of innovative solutions, examining...
-
- July 2024
- Case
“In That Crucible, You Find Innovation”: Public Safety Transformation in Albuquerque
By: Hise O. Gibson and Antonio Manuel Oftelie- July 2024
- Case
“In That Crucible, You Find Innovation”: Public Safety Transformation in Albuquerque
By: Hise O. Gibson and Antonio Manuel Oftelie -
- July 2024
- Case
Freelancer, Ltd.
By: Christopher Stanton, Karim R. Lakhani, Jin Hyun Paik and Nina CohodesOver the course of the 2010s, the rapid advancement of mobile technologies and the rise of online freelancing platforms seemed to portend a radical transformation of labor markets into on-demand, flexible talent pools. Even though several Fortune 500 companies-including Microsoft, Samsung, and General Electric-embraced digital labor solutions, enterprise adoption lagged far behind smaller businesses and startups. Despite the promising potential benefits, concerns persisted about navigating labor regulations, ensuring appropriate vetting, and guaranteeing the quality of work. Sarah Tang, the newly appointed Vice President of Enterprise at Freelancer, Ltd., took on the challenge of crafting the growth strategy, operations, and sales of Freelancer's services to Fortune 500 companies. What it would it take to convince more enterprises of the potential of on-demand freelance labor that could help them hire skilled freelancers in volume or in multiple countries simultaneously? What did the future hold for open work practices between enterprises and digital labor markets?
- July 2024
- Case
Freelancer, Ltd.
By: Christopher Stanton, Karim R. Lakhani, Jin Hyun Paik and Nina CohodesOver the course of the 2010s, the rapid advancement of mobile technologies and the rise of online freelancing platforms seemed to portend a radical transformation of labor markets into on-demand, flexible talent pools. Even though several Fortune 500 companies-including Microsoft, Samsung, and General Electric-embraced digital labor solutions,...
About the Unit
As the world of operations has changed, so have interests and priorities within the Unit. Historically, the TOM Unit focused on manufacturing and the development of physical products. Over the past several years, we have expanded our research, course development, and course offerings to encompass new issues in information technology, supply chains, and service industries.
The field of TOM is concerned with the design, management, and improvement of operating systems and processes. As we seek to understand the challenges confronting firms competing in today's demanding environment, the focus of our work has broadened to include the multiple activities comprising a firm's "operating core":
- the multi-function, multi-firm system that includes basic research, design, engineering, product and process development and production of goods and services within individual operating units;
- the networks of information and material flows that tie operating units together and the systems that support these networks;
- the distribution and delivery of goods and services to customers.
Recent Publications
The Explainability Edge? A Field Experiment on Generative AI-Augmented Evaluations of Early-Stage Innovations
- 2024 |
- Working Paper |
- Faculty Research
“In That Crucible, You Find Innovation”: Public Safety Transformation in Albuquerque
- July 2024 |
- Case |
- Faculty Research
Freelancer, Ltd.
- July 2024 |
- Case |
- Faculty Research
Research: New Hires’ Psychological Safety Erodes Quickly
- July 11, 2024 |
- Article |
- Harvard Business Review Digital Articles
Knowledge Transfer: Toyota, NUMMI, and GM
- July 2024 |
- Case |
- Faculty Research
Navigating Software Vulnerabilities: Eighteen Years of Evidence from Medium and Large U.S. Organizations
- 2024 |
- Working Paper |
- Faculty Research
Webmunk: A New Tool for Studying Online Behavior and Digital Platforms
- 2024 |
- Working Paper |
- Faculty Research
Consumer Protection in an Online World: An Analysis of Occupational Licensing
- July, 2024 |
- Article |
- American Economic Journal: Applied Economics
Harvard Business Publishing
Seminars & Conferences
There are no upcoming events.