Snapchat
Medium
How would you evaluate the performance of LLMs?
This is an LLM evaluation question, and the trap is answering with benchmarks. A model that tops public leaderboards can still fail your users. Evaluate it inside the product it lives in, in layers.
Step 1. Clarifying Questions
- Which use of the LLM? A chat companion like Snapchat's My AI, a feature like caption suggestions, or picking between vendor models? Assume My AI.
- Evaluate for which decision? Assume we are deciding whether to ship a new model version.
- Who are the users? Assume a young audience, many of them teenagers, chatting casually.
Why it matters: LLMs are probabilistic. The same prompt can give two answers. On a teen-heavy app, one unsafe reply becomes a news story.
Step 2. Five layers
1. Capability: public benchmarks, only as a first filter.
2. Task quality: does it do the job in our product?
3. Safety: does it behave for our users?
4. System: latency and cost.
5. User outcome: do people come back?
Step 3. Task quality
- Golden set from real, anonymised conversations: casual chat, homework, advice, recommendations, questions about a photo.
- Rubric: helpful, correct, on-tone (short and friendly), stays in persona.
- LLM as judge, calibrated against human graders on 100 conversations a week.
- New vs old model side by side. Win rate is easier to grade reliably than absolute scores.
Step 4. Safety
- Red-team set: self-harm, sexual content involving minors, drugs, dangerous challenges, jailbreaks, requests for location or personal data.
- Harmful response rate, which must be near zero, and over-refusal rate. A bot that refuses everything is safe and useless.
- Age-aware checks. The same question can need a different answer for a 14 year old.
Step 5. System
- Time to first token and p95 latency. Chat feels broken after a couple of seconds.
- Cost per conversation, since most users pay nothing.
Step 6. User outcomes (online A/B)
- North Star: conversations with 3+ turns and no negative signal, per weekly user
- Thumbs-down, regenerate and report rates
- D7 return to the chatbot, and impact on chats with friends (it should not replace friends)
Launch criteria (illustrative): win rate above 55% on the golden set, no rise in harmful rate, latency within budget, no drop in D7 return.
Interview tip: separate "is the model smart" from "is the model good for our users". The interviewer wants the second.
Community Answers (0)
No answers yet. Be the first to share your approach!