The Effects of AI Confidence Level Disclosure on AI Adoption, Employee Efficiency and Efficacy: A Field and Laboratory Investigation
The proposed study analyses how disclosing the confidence level of artificial intelligence (AI) influences reliance on AI advice and affects employee efficiency and efficacy. We propose conducting a randomized field experiment in a contact center of a large insurance company. Agents are randomly assigned to one of four groups: generative AI assistance; generative AI assistance with confidence level disclosure; generative AI assistance restricted to answer above a certain confidence threshold; and no AI. We will explore the impact of confidence level disclosure on call handling time as a proxy for efficiency, on customer satisfaction to proxy for efficacy, agents’ satisfaction, and AI adoption rates. Finally, we will assess the benefits of AI and AI confidence level disclosure across task and agent characteristics, including task difficulty, agent knowledge, expertise, and demographics. To further understand the effects of confidence level disclosure on reliance on algorithmic advice, we propose conducting a first laboratory experiment with a 2x3 between-subjects design, where participants receive either human or algorithmic advice, with high, low, or no confidence level. We propose a second laboratory experiment where participants choose between receiving human or algorithmic advice, with sources having either a high, low, or no confidence level disclosed.