Using a Chatbot to Assess: Application of a Conversational Tool in an Assessment Course
Keywords:
assessment, chatbot, generative AIAbstract
The authors describe the development of an AI chatbot, based on a large language model (LLM), as a formative assessment tool in a master’s-level educational assessment class. The pedagogical goal was to provide an experiential assessment and feedback on student content knowledge. The students and instructor found the conversational assessment straightforward, but were concerned about feedback quality, grading reliability, and development time. For use in classroom assessments, LLM-based chatbots and related applications will need guidance from educators on appropriate conversational responses. Educators will also need to develop verification methods regarding the reliability of chatbot-assigned grades.
Downloads
References
Adiguzel, T., Kaya, M. H., & Cansu, F. K. (2023). Revolutionizing education with AI: Exploring the transformative potential of ChatGPT. Contemporary Educational Technology, 15(3), ep429. https://doi.org/10.30935/cedtech/13152
AI for Education. (2023). GenAI chatbot prompt library for educators. AI for Education. https://www.aiforeducation.io/prompt-library
American Educational Research Association, American Psychological Association, National Council on Measurement in Education (AERA, APA, & NCME). (2014). Standards for educational and psychological testing. American Educational Research Association. https://www.testingstandards.net/open-access-files.html
Clarke, M. & Luna-Bazaldua, D. (2021). Primer on large-scale assessments of educational achievement. World Bank Publications. https://hdl.handle.net/10986/35494
Davar, N. F., Dewan, M. A. A., & Zhang, X. (2025). AI chatbots in education: Challenges and opportunities. Information, 16(3), Article 3. https://doi.org/10.3390/info16030235
Dolman, J. (2024, March 5). Do we really need AI wrappers? The AI English Teacher. https://theaienglishteacher.wordpress.com/2024/03/05/do-we-really-need-ai-wrappers/
Eke, D. O. (2023). ChatGPT and the rise of generative AI: Threat to academic integrity? Journal of Responsible Technology, 13, 100060. https://doi.org/10.1016/j.jrt.2023.100060
Fan, J., Sun, T., Liu, J., Zhao, T., Zhang, B., Chen, Z., Glorioso, M., & Hack, E. (2023). How well can an AI chatbot infer personality? Examining psychometric properties of machine-inferred personality scores. Journal of Applied Psychology, 108(8), 1277–1299. https://doi.org/10.1037/apl0001082
Gruenhagen, J. H., Sinclair, P. M., Carroll, J.-A., Baker, P. R. A., Wilson, A., & Demant, D. (2024). The rapid rise of generative AI and its implications for academic integrity: Students’ perceptions and use of chatbots for assistance with assessments. Computers and Education: Artificial Intelligence, 7, 100273. https://doi.org/10.1016/j.caeai.2024.100273
Herft, A. (2023). A Teacher’s Prompt Guide to ChatGPT aligned with ’What Works Best’. https://usergeneratededucation.files.wordpress.com/2023/01/a-teachers-prompt-guide-to-chatgpt-aligned-with-what-works-best.pdf
Huang, K. (2023, January 18). Alarmed by A.I. chatbots, universities start revamping how they teach. The New York Times, International Edition. https://www.proquest.com/docview/2766885034/citation/30EC3A82EB184DB1PQ/1
Hmoud, M., Swaity, H., Anjass, E., & Aguaded-Ramírez, E. M. (2024). Rubric development and validation for assessing tasks’ solving via AI chatbots. Electronic Journal of E-Learning, 22(6), 1–17. https://doi.org/10.34190/ejel.22.6.3292
Ifenthaler, D., Gibson, D., Prasse, D., Shimada, A., & Yamada, M. (2020). Putting learning back into learning analytics: Actions for policy makers, researchers, and practitioners. Educational Technology Research and Development. https://doi.org/10.1007/s11423-020-09909-8
Kahoot. (2024). About us. https://kahoot.com/company/
Khademi, A. (2023). Can ChatGPT and bard generate aligned assessment items? A reliability analysis against human performance. ArXiv Preprint. ArXiv:2304.05372.
Koretz, D. (2009). Measuring up: What educational testing really tells us. Harvard University Press.
Labadze, L., Grigolia, M., & Machaidze, L. (2023). Role of AI chatbots in education: Systematic literature review. International Journal of Educational Technology in Higher Education, 20(1), 56. https://doi.org/10.1186/s41239-023-00426-1
Latif, E., Mai, G., Nyaaba, M., Wu, X., Liu, N., Lu, G., Li, S., Liu, T., & Zhai, X. (2023). AGI: Artificial General Intelligence for Education (arXiv:2304.12479). arXiv. https://doi.org/10.48550/arXiv.2304.12479
Okonkwo, C. W., & Ade-Ibijola, A. (2021). Chatbots applications in education: A systematic review. Computers and Education: Artificial Intelligence, 2, 100033. https://doi.org/10.1016/j.caeai.2021.100033
Perkins et al. (2024). The AI assessment scale.
Peters, M. A. (2018). Deep learning, education and the final stage of automation. Educational Philosophy and Theory, 50(6–7), 549–553. https://doi.org/10.1080/00131857.2017.1348928
Playlab.ai. (2024). Terms of Use. https://www.playlab.ai/policies/terms-of-use
Plickers (2021). What is Plickers? https://help.plickers.com/hc/en-us/articles/360009395854-What-is-Plickers
Roose, K. (2023, January 12). Don’t Ban ChatGPT in Schools. Teach With It. The New York Times. https://www.nytimes.com/2023/01/12/technology/chatgpt-schools-teachers.html
Rudolph, J., Tan, S., & Tan, S. (2023). ChatGPT: Bullshit spewer or the end of traditional assessments in higher education? Journal of Applied Learning and Teaching, 6(1), Article 1. https://doi.org/10.37074/jalt.2023.6.1.9
Schick, A., Feine, J., Morana, S., Maedche, A., & Reininghaus, U. (2022). Validity of Chatbot Use for Mental Health Assessment: Experimental Study. JMIR mHealth and uHealth, 10(10), e28082. https://doi.org/10.2196/28082
Stahl, B. C., Schroeder, D., & Rodrigues, R. (2023). Ethics of Artificial Intelligence: Case Studies and Options for Addressing Ethical Challenges. Springer International Publishing. https://doi.org/10.1007/978-3-031-17040-9
Wang, T., Lund, B. D., Marengo, A., Pagano, A., Mannuru, N. R., Teel, Z. A., & Pange, J. (2023). Exploring the potential impact of artificial intelligence (AI) on international students in higher education: Generative AI, chatbots, analytics, and international student success. Applied Sciences, 13(11), 6716. https://doi.org/10.3390/app13116716
Williams, R. T. (2024). The ethical implications of using generative chatbots in higher education. Frontiers in Education, 8. https://doi.org/10.3389/feduc.2023.1331607
Yang, S., & Evans, C. (2020). Opportunities and challenges in using ai chatbots in higher education. Proceedings of the 2019 3rd International Conference on Education and E-Learning, 79–83. https://doi.org/10.1145/3371647.3371659
Downloads
Published
Issue
Section
Categories
License
Access Agreement Journal on Excellence in College Teaching
Before proceeding you must agree to the terms and conditions of usage as outlined below by clicking on the Accept button and/or by both parties’ signatures below. You will have to do this only once. After agreement, you will be redirected back to the main Journal page. A pdf copy of the terms is available for download.
This Access Agreement (the "Agreement") is effective upon processing of payment ("Effective Date") and is entered into by and between the Journal on Excellence in College Teaching (“JECT”) and the Customer (“Customer").
This Agreement constitutes the entire agreement and supersedes and voids all prior communications, understandings, and agreements relating to the Product(s), including any terms of use displayed to Authorized Users via the online site of the Product(s). Alterations to the Agreement and to any Addendum to the Agreement are only valid and binding if they are recorded in writing and signed by both parties
. I. Definitions
"Authorized Users" shall mean individuals who are authorized by the Customer (which shall include those individuals authorized by the Institutions hereunder) to access the Customer's information services whether on-site or off-site via secure authentication and who are affiliated with the Customer as a student (undergraduates and postgraduates), employee (whether on a permanent or temporary basis), or a contractor of the Customer. Individuals who are not a current student, employee, or a contractor of the Customer, but who are permitted to access the Customer's information services from computer terminals within the physical premises of the Customer ("Walk-In Users"), are also deemed to be Authorized Users, but only for the time they are within the physical premises of the Customer. Walk-In Users may not be given means to access the Product(s) when they are not within the physical premises of the Customer.
"Commercial Use" shall mean use for the purpose of monetary reward (whether by or for the Customer or an Authorized User) by means of the sale, resale, loan, transfer, hire, or other form of exploitation of the Product(s). For the avoidance of doubt, neither recovery of direct cost by the Customer from Authorized Users, nor use by the Customer or Authorized Users of the Product(s) in the course of research funded by a commercial organization, shall be deemed to constitute Commercial Use.
"Educational Purposes" shall mean for the purpose of education, teaching, distance learning, private study and/or research as described in Section II below.
"Institutions" shall mean the Customer's participating institutions, if applicable.
"License" shall mean the non-exclusive, non-transferable right to access and use the Product(s) pursuant to the specific terms and conditions set forth in this Agreement.
"Product(s)" shall mean the products, materials and/or information contained therein that are subject to this Agreement. Product(s) include the Journal on Excellence in College Teaching and the archive of the Learning Communities Journal.
"Reasonable Amount" shall be determined based on guidelines set forth by 17 U.S. Code § 107 (Limitations on exclusive rights, Fair use).
"Secure Authentication" shall mean access to the Product(s) by Internet Protocol ("IP") ranges or by another means of authentication agreed between the Publisher and Customer or Institutions (if applicable) from time to time.
II. Authorized Use of Product(s)
Customer, the Institutions (if applicable), and Authorized Users may use the Product(s) for Educational Purposes as follows:
Analysis. Authorized Users shall be permitted to extract or use information contained in the Product(s) for Educational Purposes, including, but not limited to, text and data mining, extraction and manipulation of information for the purposes of illustration, explanation, example, comment, criticism, teaching, research, or analysis.
Course Packs. Customer, the Institutions, and Authorized Users may use a Reasonable Amount of the Product(s) in the preparation of course packs or other educational materials.
Digital Copy. Customer, the Institutions, and Authorized Users may download and digitally copy a Reasonable Amount of the Product(s).
Display. Customer, the Institutions, and Authorized Users shall have the right to electronically display the Product(s) to the extent necessary to further the intent and purpose of this Agreement.
Electronic Reserve. Customer, the Institutions, and Authorized Users may use a Reasonable Amount of each of the Product(s) in connection with specific courses of instruction offered by Customer.
Interlibrary Loan. The Customer and the Institutions shall be permitted to use Reasonable Amounts of the Content to fulfill occasional requests from other, non-participating institutions, a practice commonly called Interlibrary Loan ("ILL"). Customer and the Institutions shall fulfill such requests in compliance with Section 108 of the United States Copyright Law (17 USC S108, "Limitations on exclusive rights: Reproduction by libraries and archives") and the Guidelines for the Proviso of Subsection 108(2g)(2) prepared by the National Commission on New Technological Uses of Copyrighted Works (CONTU).
The electronic form of the Product(s) may be used as a source for ILL. Customer and the Institutions shall include copyright notices on all ILL transmissions. Notwithstanding anything herein to the contrary, in no event shall any non-secure electronic transmission of files be permitted.
Print Copy. Customer, the Institutions, and Authorized Users may print a Reasonable Amount of the Product(s).
Recover Copying Costs. Customer and the Institutions may charge a reasonable fee to cover costs of copying or printing portions of Product(s) for Authorized Users.
Scholarly Sharing. Authorized Users may transmit to a third party colleague in hard copy or electronically, Reasonable Amounts of the Product(s) for personal use, professional use, or Educational Purposes but in no event for Commercial Use. In addition, Authorized Users have the right to use, with appropriate credit, figures, tables, and brief excerpts from the Product(s) in the Authorized User's own scientific, scholarly, and educational works.
Text Mining. Authorized Users may use the licensed material to perform and engage in text mining/data mining activities for legitimate academic research and other Educational Purposes. Those uses beyond educational use shall require permission from the Publisher.
III. Restrictions
Except as provided herein, the institution shall make reasonable efforts to inform its authorized users not to use, alter, decompile, modify, display, or distribute the Product(s) as follows:
Alter Identification. Remove, obscure, or modify copyright notices, text acknowledging, attributions, or other means of identification or disclaimers as they appear. Alter Product(s).
Alter, decompile, adapt, or modify the Product(s), except to the extent necessary to make it perceptible on a computer screen, or as otherwise permitted in this Agreement. Alteration of words or their order is strictly prohibited.
Commercial Use. No Commercial Use of the Product(s) shall be permitted unless the Customer or an Authorized User has been granted prior written consent by an authorized representative of the Product(s). Use of all or any part of the Product(s) for any Commercial Use or for any purpose other than Educational Purposes.
Distribution. Display or distribute any part of the Product(s) on any electronic network, including without limitation, the Internet, and any other distribution medium now in existence or hereinafter created, other than by a Secure Authentication; print and distribute any portion(s) of the Product(s)s to persons or entities other than the Customer or Authorized Users, except as provided in Section II.
JECT acknowledges that the Customer cannot police or control the actions of its students, faculty, and other Authorized Users with respect to their use of the Product(s). In the event of abuse, the institution shall make prompt and reasonable efforts to heal the breach and notify the publisher.
IV. Term and Termination
This agreement shall commence on the Effective Date and shall remain in effect unless and until terminated as permitted herein (the "Term"). There is no perpetual electronic access to content made available during the term of the agreement.
JECT may terminate this Agreement if Customer violates any of the terms and conditions set forth herein. In the event of any termination of access, JECT will promptly notify the Customer of the basis for termination.
The Customer may terminate this Agreement if sufficient funds are not provided or allotted in future government-approved budgets of the Customer (or reasonably available or expected to become available from other sources at the time the Customer’s payment obligation attaches) to permit the Subscriber, in the exercise of its reasonable administrative discretion, to continue this Agreement.
In the event of any unauthorized use of the Product(s) by an Authorized User, Customer shall cooperate with JECT in the investigation of any unauthorized use of the Product(s) of which it is made aware and shall use reasonable efforts to remedy such unauthorized use and prevent its recurrence. JECT may terminate such Authorized User's access to the Product(s) after first providing reasonable notice to the Customer (in no event less than two (2) weeks) and cooperating with the Customer to avoid recurrence of any unauthorized use. In the event of any termination of access, JECT will promptly notify the Customer
. V. Refunds
In the event that a subscription is canceled by the Customer prior to the subscription end date, the following will be used as guidelines for refunds.
Electronic subscriptions. The Customer shall be entitled to a full refund within 14 days of the start of the most recent subscription term. Refunds requested after 14 days but no later than 60 days from the start of the most recent subscription term will be allowed, minus a 30% processing fee. Refunds will not be granted if requested more than 60 days after the start of the most recent subscription term.
