Amazing AI

Seeing if teachers can identify ChatGPT generated paragraphs versus grade 6 student written paragraphs.
Aiden Chow
Grade 6

Hypothesis

If I give teachers a paragraph generated by ChatGPT versus a paragraph written by students, then teachers will not recognize which one was generated by ChatGPT because of the way ChatGPT phrases words and sentences to write like a human.

 

 

 

 

Research

  • OpenAI was founded on December 11, 2015 in San Francisco, California.
  • OpenAI created ChatGPT with American entrepreneurs Sam Altman (CEO) and Greg Brockman (President).
  • ChatGPT was released on November 30, 2022.
  • GPT stands for generative pretrained transformer.
  • ChatGPT had 1 million users in 5 days due to its popularity.
  • ChatGPT uses a GPT 3.5 engine but there is an upgrade to a GPT 4 engine for $20 a month.
  • It can generate up to 25,000 words and 26 languages. It may seem like a lot, but there are 217,000 words in English and about 7,000 different languages.
  • ChatGPT was tested in 7 different languages and found better at translation than Bard (Google), Bing (Microsoft) and DeepL translate (DeepL).
  • ChatGPT can write with your writing style by examining many pieces of your writing.
  • In many countries around the world there are restrictions so that AI/ChatGPT can not harm anyone.

Variables

Manipulated: 

  1. The topic of the paragraphs. 
  2. Different levels of writing.
  3. Different grade level teachers evaluating the paragraphs.
  4. ChatGPT was instructed with different prompts.

Responding: The teachers identifying the paragraphs as ChatGPT written or student written.

Controlled: 

  1. Amount of students writing the paragraphs.
  2. Amount of teachers identifying the paragraphs.
  3. Amount of paragraphs.

Procedure

  1. Gather three Grade 6 students and about 25 teachers who would like to participate in the experiment. 
  2. Pick the topics for ChatGPT and the students to write about. 
  3. Prepare instructions for the students to write their paragraphs.
  4. Ask ChatGPT to write paragraphs with the prompts. 
  5. Once you have collected all the paragraphs, create a result sheet for teachers. 
  6. Email every participating teacher the paragraphs and result sheet. 
  7. After you receive the result sheets back, document them into google sheets and analyze the data.

Observations

  1. Halfway through the experiment I noticed that a lot of teachers believed there were three ChatGPT paragraphs.
  2. On average the teachers identified 9.4/15 paragraphs correctly. 
  3. 3/15 teachers identified all the paragraphs correctly on two out of three topics and those were the highest scoring teachers.
  4. 66.67% of the teachers identified more than 50% of the paragraphs correctly.
  5. One teacher did not identify any correct paragraphs on one of the topics.
  6. Some teachers said they guessed the answer when providing me with their results.
  7. 35.7% of teachers responded.

Analysis

 


Conclusion

Teachers can correctly identify if the paragraphs are student or ChatGPT generated 66% of the time. This means that if a student incorporates ChatGPT in a way that guides their writing then the teachers will have a hard time identifying this. It also means that for teachers to determine whether a piece of writing is written by a student or ChatGPT, they have to know the student's personality, writing style, and writing skill really well. They may also have to research what a paragraph would look like if it were generated by ChatGPT.

Application

Employees and bosses could use ChatGPT in their daily lives by using it to assist them in their jobs by doing simple tasks. Students in High School and College can also use it for their homework. For teachers and instructors it can help create assignments to teach to students.

Sources Of Error

  1. Teachers could have played around with ChatGPT and asked the same questions to see its response.
  2. The amount of teaching experience each teacher has may have an effect. A teacher with limited experience may not know how students write compared to a teacher with significant amounts of experience.
  3. The different writing levels
  4. The grade level that the teacher teaches may matter because a teacher may not be familiar with Grade 6 level of writing.
  5. Teachers did not know what grade the students who wrote the paragraphs were in.

Acknowledgement

Thank you to the many teachers and students who participated