ChatGPT Study

Citation Author(s):
Natasha
Randall
Submitted by:
Natasha Randall
Last updated:
Fri, 09/01/2023 - 04:41
DOI:
10.21227/4rxb-zv06
License:
Creative Commons Attribution
425 Views
Categories:
Artificial Intelligence
Education
Keywords:
artificial intelligence, ChatGPT, software engineering education, curriculum development
0
0 ratings - Please login to submit your rating.
ACCESS DATASET CITE SHARE/EMBED

Abstract 

This dataset comprises data created during research on AI-generated code, with a focus on software engineering use-cases. The purpose of the research was to investigate how AI should be integrated into university software engineering curricula. The seven core areas of the research include: An analysis of AI and student-written Java code, an analysis of AI and human-written Python code, the results of predictions made “by eye” to distinguish between AI and student-written code, experiments to determine the capabilities of ChatGPT, individual students' feedback on experiences with ChatGPT, a set of guidelines and detailed experiences with AI tools, and the results of a survey of students regarding the use of AI tools.

Instructions: 

The seven core areas of this research are contained in each of the seven folders, and are described below.

Analysis of AI and Student-written Code: AI chatbots (ChatGPT-4, ChatGPT-3.5, Bing Chat and Bard) were used to generate code solutions to Java programming tasks (milestone assignments) taken from the 2021 presentation of the Software Engineering 2 Bachelor's course at TH Köln. Student solutions to the same assignments were stored anonymously. The differences between the AI and human written code solutions to "milestone 0" are analysed, and a simple classification model was trained to distinguish between AI and human written code. For the analysis, two approaches are compared: representing the code using manually defined features, and by OpenAI's text embedding vectors.

AI or Student-written Code Predictions: In order to determine how effectively AI-written code can be detected "by eye", a mixed set of AI- and human-written code solutions to the "Software Engineering 2" 2021 milestone 0 assignment was anonymised and given to two faculty members, who made predictions as to whether each solution was written by either a student or AI chatbot.

Analysis of AI and Human-written Python Code: ChatGPT was used to generate Python code solutions to the HumanEval problem set. The differences between the AI and human-written canonical code solutions are analysed, and a model is trained to distinguish between the two classes. The feature and embedding representation approaches are compared.

ChatGPT Capabilities Experiment: An evaluation of the capabilities of ChatGPT at completing software engineering university course assignments. By prompting ChatGPT-4 with only the original task description and provided document comments, purely AI-written code was generated as the solution to the "Software Engineering 2" 2023 assignment, comprising of the creation of a complex eCommerce system.

Student Experiences with ChatGPT: Feedback from two students on their experiences with using ChatGPT during their software engineering studies. Each use-case is evaluated, and the conversation with ChatGPT is provided.

AI Tools Experiences and Guidelines: Our experiences with generating code using different chatbots, an evaluation of their strengths and weaknesses, and a set of guidelines for using AI tools to write code.

Student Survey: The results from a survey distributed to university students. Questions relate to general AI tool use, AI use within education, and AI use specifically for coding and software engineering.

Dataset Files

LOGIN TO ACCESS DATASET FILES

QUESTIONS?