ChatGPT, an advanced language model, has been making waves in the media, with many speculating about its potential for massive disruption, particularly in the fields of accounting and auditing. However, prior research has raised concerns about ChatGPT’s performance on student assessment questions. In order to shed light on this issue, a recent research project aimed to assess the capability of newly released ChatGPT models in passing major accounting certification exams, including the CPA, CMA, CIA, and EA exams. The research paper finds that ChatGPT“ likely will prove disruptive to the accounting and auditing industries“.
The research findings demonstrate that the original ChatGPT 3.5 model performs similarly to the Wood et al. (2023) test using accounting assessment questions, with an average score of 53.1 percent across all exam parts. However, upgrading to the new ChatGPT 4 model significantly improves scores by 16.5 percent on average. Implementing few-shot training further enhances scores by an additional 6.6 percent, and allowing ChatGPT to react and reason boosts scores by an additional 8.9 percent. Combining these improvements, ChatGPT 4, with few-shot training and the ability to react and reason, achieves an impressive average score of 85.1 percent across all content tested. This updated performance demonstrates that ChatGPT has the capability to easily pass multiple-choice sections of accounting certification exams such as CPA, CMA, CIA, and EA.
Find out more about the research project here.