A Testing Framework for AI Linguistic Systems (testFAILS)

Y. Kumar, P. Morreale, P. Sorial, J. Delgado, J. Jenny Li, P. Martins

Research output: Chapter in Book/Report/Conference proceedingConference contributionpeer-review

3 Scopus citations

Abstract

This paper introduces testFAILS, an innovative testing framework designed for the rigorous evaluation of AI Linguistic Systems, with a particular emphasis on various iterations of ChatGPT. Leveraging orthogonal array coverage, this framework provides a robust mechanism for assessing AI systems, addressing the critical question, 'How should we evaluate AI?' While the Turing test has traditionally been the benchmark for AI evaluation, we argue that current publicly available chatbots, despite their rapid advancements, have yet to meet this standard. However, the pace of progress suggests that achieving Turing test-level performance may be imminent. In the interim, the need for effective AI evaluation and testing methodologies remains paramount. Our research, which is ongoing, has already validated several versions of ChatGPT, and we are currently conducting comprehensive testing on the latest models, including ChatGPT-4, Bard, Bing Bot, and the LLaMA model. The testFAILS framework is designed to be adaptable, ready to evaluate new bot versions as they are released. Additionally, we have tested available chatbot APIs and developed our own application, AIDoctor, utilizing the ChatGPT-4 model and Microsoft Azure AI technologies.

Original languageEnglish
Title of host publicationProceedings - 5th IEEE International Conference on Artificial Intelligence Testing, AITest 2023
PublisherInstitute of Electrical and Electronics Engineers Inc.
Pages51-54
Number of pages4
ISBN (Electronic)9798350336290
DOIs
StatePublished - 2023
Event5th IEEE International Conference on Artificial Intelligence Testing, AITest 2023 - Athens, Greece
Duration: 17 Jul 202320 Jul 2023

Publication series

NameProceedings - 5th IEEE International Conference on Artificial Intelligence Testing, AITest 2023

Conference

Conference5th IEEE International Conference on Artificial Intelligence Testing, AITest 2023
Country/TerritoryGreece
CityAthens
Period17/07/2320/07/23

Keywords

  • AI Linguistic Systems Testing Framework (testFAILS)
  • AIDoctor
  • Bot Technologies
  • Chatbots
  • Validation of Chatbots

Fingerprint

Dive into the research topics of 'A Testing Framework for AI Linguistic Systems (testFAILS)'. Together they form a unique fingerprint.

Cite this