1. Variations in Scoring Ranges

First of all, we explore how different score ranges might impact scoring. In particular, we wonder if the relative scores will shift, or if chatGPT will use more categories when provided with a wider range of scores. To examine this, we consider three prompts, each with a different maximum score:

• prompt1: “Evalute the overall quality of the following essay out of 5.”
• prompt2: “Evalute the overall quality of the following essay out of 7.”
• prompt3 : “Evalute the overall quality of the following essay out of 10.”

2. Role-Specific Prompts and Targeted Scoring Ranges

To investigate how ChatGPT’s scoring is influenced by the age-specific role of the essay grader (elementary, college, or graduate), we employe three distinct prompts, each assigning ChatGPT the role of an essay grader for a specific age group while maintaining a consistent 7-point scoring scale:

• Prompt 4: “You are an essay grader for elementary school students. Evaluate the overall quality of the following essay out of 7.”
• Prompt 5: “You are an essay grader for college students. Evaluate the overall quality of the following essay out of 7.”
• Prompt 6: “You are an essay grader for graduate students. Evaluate the overall quality of the following essay out of 7.”

3. Variations in Specific Scoring Criteria

Next, we incorporate specific scoring criteria into the prompts to evaluate ChatGPT’s sensitivity to different aspects of essay quality, and to investigate whether it genuinely considers the specified criteria when assigning scores:

• Prompt 7: “Evaluate the overall quality of the following essay on the use of iPads in school, considering structure, content, argument strength, and writing quality. Use a 7-point scale with higher scores indicating greater quality.”
• Prompt 8: “Evaluate the overall quality of the following essay on the use of iPads in school, considering grammar, coherence, organization, and relevance. Use a 7-point scale with higher scores indicating greater quality.”
• Prompt 9: “Evaluate the overall quality of the following essay on the use of iPads in school, considering development of ideas, organization, and language facility and convention. Use a 7-point scale with higher scores indicating greater quality.”

4. Guided Prompts: Defining Outcomes

To assess whether ChatGPT’s evaluation is sensitive to the specified output format, we designe three prompts that are identical in their core request (to evaluate the overall quality of an essay on the use of iPads in school using a 7-point scale), but differ in how the response is to be presented:

• Prompt 10: “Evaluate the overall quality of the following essay regarding the use of iPads in school. Use a 7-point scale with higher scores indicating greater quality. Present your response as a single numeric score.”
• Prompt 11: “Evaluate the overall quality of the following essay regarding the use of iPads in school. Use a 7-point scale with higher scores indicating greater quality. Format your response as a JSON object with ‘score’ as the key.”
• Prompt 12: “Evaluate the overall quality of the following essay regarding the use of iPads in school. Use a 7-point scale where the descriptors range from ‘Very Poor’ to ‘Excellent.’ Provide your response as either ‘Very Poor’, ‘Poor’, ‘Below Average’, ‘Average’, ‘Above Average’, ‘Very Good’, or ‘Excellent.’”

5. Advanced Grading Techniques: Few-Shots Learning and Chain-of-Thought Reasoning

We wrap up by using some more advanced ChatGPT guidance techniques to see how they help with grading. In particular, we investigate how ChatGPT’s responses are affected by incorporating few-shot learning and chain-of-thought reasoning into the prompts. Before we show you the prompts, we quickly describe that these essential terms from the realm of LLMs mean to ensure a clearer understanding for everyone:

Prompts: Guiding LLM Responses

• Zero-shot prompting: This involves providing a basic task description without any examples. The LLM relies solely on its pre-existing knowledge and language understanding to generate a response.
• Few-shot prompting: This includes a few examples in the prompt to help guide the LLM’s response. The LLM learns from these examples and uses them to generate more relevant and accurate outputs.

Chain-of-Thought Reasoning

• Chain-of-thought reasoning (CoT) is a technique that enhances the reasoning capabilities of LLMs. It involves breaking down complex problems into a series of smaller, more manageable steps, similar to how humans approach problem-solving. By generating intermediate reasoning steps, CoT allows LLMs to tackle intricate tasks that require logical deduction and multi-step reasoning.

Previous studies (e.g., Wu et al., 2023) have indicated that both few-shot prompting and COT reasoning can improve the accuracy and reliability of LLMs. These approaches have been shown to be effective in various domains, leading to more sophisticated and human-like responses from LLMs.

A Note on “Base” in Essay Grading

• In the context of essay grading, there is no truly “zero-shot” approach. To obtain relevant scores, the prompt for essay grading must include instructions describing how to assess the essays on a specific scale and using a specific rubric. We refer to this comprehensive prompt as the “base prompt,” though it’s not a standard term in the field.

Ok, armed with above, we tried the following three prompts:

• Prompt 13: “Evaluate the overall quality of the following essay regarding the use of iPads in school based on the following 3 criteria: (1) Development of Ideas, measuring the depth, complexity, and richness of details and examples; (2) Organization, focusing on the logical structure, coherence, and overall focus of ideas; (3) Language Facility and Convention, evaluating clarity, effectiveness in sentence structure, word choice, voice, tone, grammar, usage, and mechanics. Use a 7-point scale with higher scores indicating greater quality. Present your response as a numeric score.”

• Prompt 14: “Evaluate the overall quality of the following essay regarding the use of iPads in school based on the following 3 criteria: (1) Development of Ideas, measuring the depth, complexity, and richness of details and examples; (2) Organization, focusing on the logical structure, coherence, and overall focus of ideas; (3) Language Facility and Convention, evaluating clarity, effectiveness in sentence structure, word choice, voice, tone, grammar, usage, and mechanics.
  
  {fewshot_examples}
  
  Use a 7-point scale with higher scores indicating greater quality. Present your response as a numeric score.”
  
  
• Prompt 15: “Evaluate the overall quality of the following essay regarding the use of iPads in school based on the following 3 criteria: (1) Development of Ideas, measuring the depth, complexity, and richness of details and examples; (2) Organization, focusing on the logical structure, coherence, and overall focus of ideas; (3) Language Facility and Convention, evaluating clarity, effectiveness in sentence structure, word choice, voice, tone, grammar, usage, and mechanics.
  
  {fewshot_examples}
  
  Use a 7-point scale with higher scores indicating greater quality. When evaluating and scoring the given essay, consider the three criteria (Development of Ideas, Organization, and Language Facility and Convention) and the few-shot examples. Present your response as a numeric score.”
  
  

(Few-shot examples can be found in the appendix.)

References

• Kim, Y., Mozer, R., Miratrix, L., & Al-Ademi, S. (2024). ChatGPT vs. Machine Learning: Assessing the Efficacy and Accuracy of Large Language Models for Automated Essay Scoring (in preparation).
• Wu, T., He, S., Liu, J., Sun, S., Liu, K., Han, Q.-L., & Tang, Y. (2023). A brief overview of ChatGPT: The history, status quo and potential future development [Conference Name:IEEE/CAA Journal of Automatica Sinica]. IEEE/CAA Journal of Automatica Sinica, 10 (5), 1122–1136. https://doi.org/10.1109/JAS.2023.123618

Appendix (Essay Examples for Few-shot Approach)

fewshot_examples = """            
   <Few-shot Examples>
   Text1: I think that those kids are mine were are 
          that parents are worried about other kids online now. Many.
   Overall Quality - 1

   Text2: yes because you can study more and do the homeworks 
          like math and science for science test. 
   Overall Quality - 2

   Text3:  I don't like the Ipads, they're too easy to break. 
           And I don't use it. So it's kind a useless 
           We have used them only maybe once or twice.  
           Also when we do use them it's only for a couple of minutes. 
So I think that they are not really necessary to learn. 
I think that you can do everything you can do on a Ipad on a computer. 
Overall Quality - 3

Text4:  The reason why you should use Ipad 
because you can use it to help you with school. 
And help with word with online dictionaries. 
But the that post mean videos will lose the Ipad for one or two weeks. 
It depend how bad they make the video be. 
If they don't stop doing 
           it can lead to getting suspended or lose the Ipad 
           for a long time. And by using the Ipads 
           it will make the students very smart. 
           And will the student understand more thing in a advanced way. 
           While they're using their Ipads for education 
they can have fun with the Ipads to play math games.
And in the morning you can play games on the Ipad. 
It will be good to have Ipads in the school.
Overall Quality - 4

Text5:  I believe that Ipads should be used in school because 
when kids get older they should be able to know how to use it
because when kids get older a lot of jobs use Ipads or tablets. 
Also I believe that you can find things for learning 
more online than on books and on paper. 
just because some kids do stupid things 
online doesn't mean all the kids should suffer. 
           Also a lot of things are not on books anymore than <what is> online. 
           Some kids have a bad spelling problem 
           and on a tablet or Ipad have work check and it helps a lot. 
           Also it make school more fun for kids, 
           I know if I didn't like school we had Ipads or tablets to do work. 
It would make me want to learn. 
that's why I believe we should have Ipads or tablets in our school .  
   Overall Quality - 5

   Text6:  In my opinion I believe 
           that students should totally be able to use the Ipod or Ipad. 
           My reason for this statement is 
           because I believe that it is a better way to learn. 
           Now in days we have Google_Chrome 
           and kids can basically find all they need on the Internet. 
           Keep in mind that it two thousand fourteen 
           we have Ipads and tablets so why not use it.
           Also you may even play cool math games and educational games. 
           Another reason why is because kids can also find words 
           and use words they have never seen on the Internet.     
           it's basically a fun but cool way. 
Not only that but kids now and 
days just would prefer the Ipod or Ipad over the computer. 
So just think about it also the computer is just as bad really 
because either way you can still post negative comments. 
Also to solve the problems about negative comments 
you can just simply block websites like Facebook, Instagram, 
and Vine so that the people's comments cannot stop the other kids 
           from having fun and learning on top of that. 
           Why should one group of kids stop the other kids from learning. 
           So yes I disagree.  
   Overall Quality - 6

   Text7:  I believe all students should have 
           the opportunity to use Ipads while in school.  
           Without these electronic devices some students 
           may have trouble finding access 
           to other electronic devices 
           in order to complete homework and school work. 
           I also believe that Ipads will be beneficial in a classroom 
           because it will let students research topics 
           and that research may be needed for an in school project. 
           These are some of the pros to having access to Ipads during school. 
           Although there are many pros to the Ipads there are also a few cons.  
           Such as the Ipad being distracting for students.   
           The students with Ipads may be playing games 
           or looking things up on the Internet 
           that has nothing to do with the classroom topic. 
           Also the students may be posting harsh comment geared 
           towards other students on social media websites during class or at home. 
           All of these problem can be fixed easily. 
           To cut down on the number of students playing games 
           during class just have the student keep the Ipads off and in their bags 
           or under their seat until the teacher instructs them 
           to take them out and use the Ipad for a certain purpose. 
           This reduce the number of cruel comments being posted during class. 
           Another solution to stop mean comments from going viral at home is 
           to have the students return the Ipads to a cart 
           at the end of the day and receive them again in the morning. 
           Overall there are pros and cons to classroom Ipads. 
           But the cons can be fixed with simple rules. 
           This is way I believe the Ipads are an asset to the classroom.
   Overall Quality - 7
   """