The research team systematically evaluated the performance of top-tier AI models, including Google Gemini, ChatGPT-4, Claude, Mixtral, and Llama-2b on multiple-choice questions (MCQs) sourced directly from the Becker CPA exam preparation suite.

Key findings include:

• LLMs Show Promise: ChatGPT-4 and Claude Opus remarkably passed multiple sections of the CPA exam, displaying proficiency in both auditing and financial reporting.
• Human Accountants are Not Obsolete…Yet: Despite the impressive performance of some LLMs, human accountants are not likely to be replaced as current AI models cannot replicate the full range of skills and judgment required for all areas of accounting. AI excelled in highly structured tasks with definitive answers, but struggled with complex, open-ended questions.
• Model Strengths and Weaknesses: ChatGPT-4 dominated in financial reporting and regulation, while Claude Opus was strongest in auditing. Llama-2b significantly underperformed, indicating clear needs for further development in accounting-specific applications.

“This landmark study provides a sobering assessment of the true potential of AI in accounting,” says Will Zacher, lead author of the paper. “While AI models have surpassed our initial expectations in certain respects, this field remains in its infancy. AI tools, carefully deployed, may soon be able to handle a significant portion of routine accounting tasks, but the need for skilled human accountants to interpret results and provide strategic guidance remains paramount.”

The team rigorously tested each AI model on three independent runs across all CPA exam sections (AUD, BEC, FAR, REG). The criterion for success was a passing score (75%).