Conversation Analysis Guide

A step-by-step guide for teachers to efficiently evaluate AI tutor conversations.

Quick Analysis Process (5-10 minutes per conversation)

Step 1: Initial Scan (1-2 minutes)

• Length Check: Is the conversation substantive? (Aim for 10+ exchanges)
• Engagement Check: Does the student participate actively?
• Topic Focus: Does the conversation stay on the assigned topic?

Step 2: Question Assessment (2-3 minutes)

Look for these types of questions from the student:

High-Quality Questions (give full credit):

• “Why does this work this way?”
• “What would happen if I changed [X]?”
• “How does this connect to [previous topic]?”
• “Can you help me understand the reasoning behind this?”
• “What are some other ways to approach this?”

Medium-Quality Questions (partial credit):

• “Can you explain this step again?”
• “I don’t understand [specific concept]”
• “Is this the right answer?”
• “What’s the next step?”

Low-Quality Questions (minimal credit):

• “What’s the answer?”
• “Just tell me how to do it”
• “Is this right?” (without showing work)

Step 3: Critical Thinking Assessment (2-3 minutes)

Look for evidence of:

• Analysis: Breaking down problems into components
• Synthesis: Connecting ideas across different parts of the topic
• Evaluation: Judging the validity of different approaches
• Application: Using learned concepts in new situations

Red Flags:

• Student immediately asks for answers without attempting to think
• No follow-up questions or curiosity
• Responses show copy-paste behavior
• Student doesn’t engage with AI’s guiding questions

Step 4: Learning Progression (1-2 minutes)

Track whether the student:

• Shows improvement in understanding over the conversation
• Begins to answer their own questions
• Makes connections to previous learning
• Demonstrates growing confidence with the material

Detailed Evaluation Criteria

Question Quality Rubric

Excellent (4 points)

• Questions demonstrate deep thinking and genuine curiosity
• Student builds on previous responses to ask more sophisticated questions
• Questions seek to understand underlying principles, not just procedures
• Evidence of metacognitive awareness (“I think I understand X, but I’m confused about Y”)

Good (3 points)

• Questions show student is thinking about the material
• Some evidence of building on previous responses
• Asks for clarification when genuinely confused
• Shows some curiosity beyond immediate problem-solving

Satisfactory (2 points)

• Questions are relevant to the topic
• Student asks for help when stuck
• Basic level of engagement with the material
• Questions focus mainly on procedure rather than understanding

Needs Improvement (1 point)

• Questions seek immediate answers without thinking
• No evidence of building on previous responses
• Focus solely on getting the “right answer”
• Minimal engagement with the learning process

Critical Thinking Indicators

Strong Evidence:

• Student challenges or questions the AI’s explanations
• Makes connections to other mathematical concepts or real-world applications
• Proposes alternative solution methods
• Recognizes and corrects their own mistakes
• Asks “what if” questions to explore edge cases

Moderate Evidence:

• Student explains their thinking process
• Attempts to verify answers through different methods
• Asks follow-up questions to deepen understanding
• Shows awareness of when they don’t understand something

Weak Evidence:

• Student follows directions without questioning
• Focuses on memorizing steps rather than understanding concepts
• Doesn’t attempt to verify or check their work
• Shows little curiosity about why methods work

Common Conversation Patterns

Positive Patterns

1. The Explorer: Asks many “why” and “what if” questions
2. The Connector: Links new concepts to previously learned material
3. The Skeptic: Questions explanations and seeks verification
4. The Improver: Shows clear progression in understanding throughout the conversation

Concerning Patterns

1. The Answer Seeker: Only asks for final answers
2. The Passive Receiver: Rarely asks questions, just acknowledges
3. The Copy-Paster: Shows work that seems copied without understanding
4. The Topic Jumper: Doesn’t engage deeply with any single concept

Feedback Templates

For High-Quality Conversations

“Excellent work! Your questions showed deep thinking, especially when you asked [specific example]. I particularly liked how you connected [concept A] to [concept B]. This type of curiosity and critical thinking will serve you well in mathematics.”

For Medium-Quality Conversations

“Good effort in your conversation with the AI tutor. I noticed you asked helpful clarifying questions about [specific topic]. To enhance your learning, try asking more ‘why’ questions to understand the reasoning behind the steps, not just the steps themselves.”

For Low-Quality Conversations

“I can see you completed the AI tutor conversation, but I’d like to see more engagement with the learning process. Instead of asking for direct answers, try asking questions like ‘Why does this method work?’ or ‘What would happen if I tried a different approach?’ This will help you develop stronger problem-solving skills.”

Integration with Existing Assessments

This analysis can be used to:

• Supplement traditional homework grades (10-20% of assignment value)
• Inform classroom instruction (identify concepts that need more coverage)
• Track student growth (compare conversation quality over time)
• Identify students who need additional support (those showing concerning patterns)
• Recognize students demonstrating exceptional critical thinking (those showing positive patterns)

Time-Saving Tips

1. Use consistent evaluation sessions: Review multiple conversations at once
2. Focus on patterns: Look for recurring behaviors rather than isolated incidents
3. Sample selectively: For large classes, analyze every 3rd or 4th conversation initially
4. Use audio notes: Speak your observations rather than writing them
5. Create shortcuts: Develop abbreviations for common observations
6. Batch similar topics: Review all algebra conversations together, geometry conversations together, etc.

Remember: The goal is to encourage and develop critical thinking, not to penalize students for their current level of questioning ability.