How to Measure CSAT When AI Handles Your Support: Tools and Survey Questions
To measure CSAT when AI handles your support, report it separately for chats the AI resolved, chats it handed to a person, and chats people handled alone. Pair the survey with a resolution question, and score the conversations customers never rate.
This guide covers the formula, the survey questions to ask, the tools that support it and a six-step setup.

1. The Short Answer
CSAT (customer satisfaction score) is the share of survey responses that are positive, usually a 4 or 5 on a five-point scale. When an AI agent handles part of your support, one blended CSAT number stops being useful. It mixes chats the AI solved with chats it passed to a person, and it only reflects the customers who answered the survey.
The fix has three parts. Tag every conversation by who resolved it. Ask whether the problem was solved, not only how the customer felt. Then use transcript scoring, where AI reads each chat and rates it, to cover the conversations nobody rated.
Key takeaways
- Report CSAT separately for AI-resolved, AI-to-human and human-only conversations. A blended score hides where problems are.
- Ask “Did we solve your problem?” next to the satisfaction question. Satisfaction without resolution can mislead.
- Surveys only cover customers who reply. AI-scored metrics such as Intercom’s CX Score rate every conversation.
- Check AI-scored ratings against real survey answers and human review before you rely on them.
- Track repeat contacts and escalations alongside CSAT. Klarna reported repeat inquiries next to its satisfaction score.
2. Why Does Standard CSAT Break With AI Support?
Traditional CSAT assumes a person handled the ticket. You send a survey when it closes and average the answers. AI customer service changes three things.

- Fewer people answer: Surveys capture only the customers who choose to respond. Intercom’s documentation notes that CSAT relies on a small, self-selecting group. As AI handles more chats, fewer conversations get any human review at all.
- The score gets blended: One number mixes simple chats the AI solved with hard ones it escalated. A good month on order-status questions can hide a bad month on refunds.
- Calm is not the same as solved: A customer can rate a friendly bot highly and still contact you again tomorrow. Without a resolution question, you can’t tell which happened.
The stakes are real. The Qualtrics 2026 Consumer Experience Trends Report, based on a Q3 2025 survey of more than 20,000 consumers, found nearly 1 in 5 who used AI for customer service saw no benefit. That failure rate was almost four times higher than for AI use in general.
Real example · Klarna
In February 2024, Klarna said its AI assistant handled two-thirds of its service chats in its first month and was on par with human agents on customer satisfaction. The same release reported a ~25% drop in repeat inquiries.
The release gave one overall comparison, not a breakdown by type of request. Note that Klarna paired satisfaction with a second outcome metric, repeat inquiries. That pairing is the habit worth copying.
3. How Do You Calculate CSAT for AI Conversations?
The formula stays the same. What changes is how you slice it.

Formula: CSAT = positive ratings (4 or 5) ÷ all ratings received × 100
The denominator is responses received, not surveys sent. Always report the response rate next to the score.
Calculate it separately for each resolution path. Each one answers a different question.

- AI only — What its CSAT tells you: Whether the AI answered correctly and the customer was happy without a person. If it drops, check: Wrong or outdated answers, missing help articles, actions the AI couldn’t complete.
- AI → person — What its CSAT tells you: How well the hand-off worked, and whether the AI kept a customer too long before passing them on. If it drops, check: Customers repeating themselves, slow transfers, missing context for the agent.
- Person only — What its CSAT tells you: Your baseline. Use it to judge whether the AI paths are better, equal or worse. If it drops, check: Staffing, queue times, agent coaching.
Note: In our view, the AI-to-human path deserves the closest watch. A low score there usually means the customer had to repeat themselves after the hand-off. Our guide to chatbot handoff and context transfer covers how to fix that.
4. Which Survey Questions Should You Ask?
Keep the survey to two questions, three at most. Ask the CSAT question and a resolution question every time. Add one more only when the conversation path calls for it.

- Core CSAT (Every conversation): How satisfied are you with the help you got today? (1–5) — Your headline number. Count 4 and 5 as positive. Keep the wording the same across AI and human chats so the scores compare fairly.
- Resolution (Every conversation): Did we solve your problem? (Yes / No / Partly) — Separates a pleasant chat from a solved one. Customers who answer “No” are your best list of AI gaps to fix.
- Effort (AI-only chats): How easy was it to get what you needed? (1–5) — Shows whether the AI made the customer work hard, such as rephrasing the question or clicking through options.
- Hand-off (After a transfer): Did you have to repeat any information after you were connected to our team? (Yes / No) — Shows where context gets lost between the AI and the agent, so you know which hand-offs to fix.
- Open text (After a low score): What could we have done better? — Optional, and only after a 1–3 or a “No”. The answers tell you which article, policy or flow to fix.
Tip: Send chat surveys inside the chat as it ends, not by email a day later. The customer still remembers the conversation, and you can tie the answer to the exact transcript.
5. Which Tools Measure CSAT for AI Customer Support?
Most AI customer support software now does one of two things. It sends a survey after the AI conversation, or it uses AI to score the transcript itself. Here is what three common platforms document, plus the custom option.

- Zendesk — How it measures AI conversations: Sends a CSAT survey on solved tickets, by email or inside messaging. Its AI agent guidance lists user satisfaction, resolution, escalation and repeat contact rates as metrics to track. Watch for: Report AI-agent and human tickets separately; one combined score blends both.
- Intercom (Fin) — How it measures AI conversations: Survey-based CSAT plus CX Score, which uses AI to rate conversations 1–5 without a survey. Ratings of 4 or 5 count as positive, and you can segment by agent type. Watch for: CX Score needs Fin and the Pro add-on. It is an AI estimate, not the customer’s own rating.
- Zendesk AI agents (Advanced) with Freshchat — How it measures AI conversations: Triggers the CSAT survey when the AI resolves the conversation, or 15 minutes after the chat goes inactive. Watch for: The AI agent needs its own agent account so ratings are attributed to it, not to a person.
- Custom build — How it measures AI conversations: Survey through your help desk API, then score transcripts with a language model against your own rubric in your data warehouse. Watch for: Most control, but you own the setup and must validate the scoring against human review.
Sources: Zendesk CSAT docs, Zendesk AI agent metrics, Intercom CX Score docs, Zendesk AI agents for Freshchat docs. Features as documented in 2025–2026; check current plans before buying.
AI-scored metrics solve the coverage problem, but they are a model’s judgement, not the customer’s. In our view, you should check them against real survey answers and a sample of human reviews before you trust them on their own.
6. Survey CSAT or AI-Scored CSAT?
Use both. They measure different things, and each covers the other’s blind spot.

Survey CSAT
- The customer’s own words and rating
- Usually covers only the customers who reply
- Can be skewed by which customers choose to answer
- Best used as ground truth for checking other scores
AI-scored CSAT
- A model’s judgement of the full transcript
- Can cover every conversation, including short ones
- Explains each score, which helps you find the cause
- Needs regular checks against surveys and human review
Want CSAT you can trust for your AI support? We can help you set it up.
7. How Do You Set It Up? 6 Steps

- Tag every conversation by path. AI only, AI then person, or person only. Most help desks can do this with a tag or a field when the AI hands off.
- Give the AI agent its own identity. A separate agent account or bot ID makes sure its ratings never mix with your team’s.
- Add the two core questions. The satisfaction question and “Did we solve your problem?” on every chat, sent inside the conversation as it ends.
- Add path-specific questions. The effort question for AI-only chats and the repeat-information question after a hand-off.
- Turn on transcript scoring. Use your platform’s AI scoring or a custom model, then compare it with survey answers each month.
- Review weekly by ticket reason. Read the low-scoring chats, fix the answer or the flow, and check that repeat contacts fall.
8. What Should You Track Next to CSAT?
CSAT alone can look healthy while customers give up quietly. Read it alongside these numbers, each split by ticket reason.

- Response rate: Share of surveyed conversations that got a rating. A high CSAT on a small response rate tells you little.
- Resolution rate: Conversations the customer confirms as solved, or with no repeat contact within seven days.
- Repeat contact rate: Customers who come back about the same issue within seven days, on any channel.
- Escalation rate: Share of AI conversations passed to a person. Too low can mean customers are stuck; too high means the AI isn’t helping.
- Time to reach a person: How long it takes a customer who asks for a human to get one.
For starting targets, Zendesk’s guidance for its AI agents suggests aiming for user satisfaction of 4.0 or more out of 5 and an escalation rate of ~20–40%. Treat these as a first reference. Your own baseline from human-handled tickets is the better comparison.
9. Frequently Asked Questions
How do you measure CSAT for an AI chatbot?
Send a short survey inside the chat as it ends, and make sure ratings are attributed to the AI, not to your agents. Report the score separately for chats the AI resolved and chats it handed off. Add a resolution question, and use transcript scoring to cover customers who don’t reply.
What survey questions should you ask after an AI support chat?
Ask “How satisfied are you with the help you got today?” on a 1–5 scale and “Did we solve your problem?” every time. Add an effort question for AI-only chats and “Did you have to repeat any information?” after a hand-off. Ask for comments only after a low score.
Is AI-generated CSAT accurate?
It can be useful, but it is a model’s estimate, not the customer’s rating. Intercom’s CX Score, for example, rates conversations 1–5 using signals such as answer quality and customer effort. Compare AI scores with real survey answers and a sample of human reviews before relying on them.
What is a good CSAT score for AI customer service?
Zendesk’s guidance for its AI agents suggests aiming for user satisfaction of 4.0 or more out of 5. A better test is your own data: compare CSAT on AI-resolved chats with CSAT on human-handled chats for the same ticket reasons, and check repeat contacts for both.
Should AI and human CSAT be reported separately?
Yes. A combined score can hide problems in one path. Klarna reported in 2024 that its AI assistant was on par with human agents on satisfaction, a comparison that is only possible when the two are measured separately. Also report the AI-to-human hand-off path on its own.
10. Conclusion
To measure CSAT when AI handles your support, split the score by who resolved each chat, add a resolution question, and score the conversations customers don’t rate. Then check every number against repeat contacts. Your next step: tag last month’s conversations by path and compare the three CSAT figures side by side.
If you’d like help putting this in place, see our AI customer support automation service.
Author




