Evaluating the Performance of ChatGPT in Accounting and Auditing Exams: An Experimental Study in North Macedonia

Abstract

In the ever-evolving landscape of education, the integration of artificial intelligence (AI) has emerged as a transformative force, reshaping conventional pedagogical methods and assessment approaches. The domains of accounting and auditing, integral for preparing future professionals, have not remained immune to this technological revolution. As AI, including ChatGPT, gains prominence, it offers the potential to enhance learning and examination experiences within these disciplines. This paper presents a pioneering North Macedonian experimental research evaluating the performance of ChatGPT's effectiveness in exam performance in accounting and auditing. Leveraging 11 subject exams comprising a total of 401 questions, this research assesses ChatGPT's proficiency in addressing intricate, context-specific questions within these domains. ChatGPT 3.5 demonstrated its reliability as a "student" by successfully passing 8 out of the 11 subjects, achieving a pass rate of 73%. The findings reveal that ChatGPT 3.5 excels in responding to qualitative open-ended questions, true/false assessments, and multiple-choice questions with a single correct answer, effectively streamlining routine tasks and offering comprehensive explanations. However, it faces challenges when dealing with quantitative calculations and complex multiple-choice questions with multiple correct answers. This study contributes to the ongoing discourse on AI's role in education, sparking discussions about student access to AI tools, the balance between memorization and referencing, the potential for AI to enhance learning experiences, and the unique contributions of educators that surpass AI capabilities. As AI continues to disrupt traditional practices across professions, including education, its responsible integration promises to enhance the educational process and deliver substantial value. ChatGPT, as a versatile AI language model, stands as a powerful tool in this endeavor, offering educators and students the potential to enrich their learning journeys while recognizing the need for judicious application within its defined constraints.