AI Quality & Evaluation Lead
Listed by employer · Checked
Posted within 60 days ·
- Work arrangement
- remote
- Employment
- Temporary
- Stated compensation
- Not specified
- Stated sponsorship
- Not stated
Does the location policy fit?
Choose where you would work from. We’ll compare it with the posting’s stated requirements.
Choose a country to check geographic compatibility. The source requirements are available below.
This checks the posting’s stated requirements, not your legal right to work. Confirm details with the employer.
Requirements
2 with evidence7 unclear or unstated- Remote policy
- remote
“Provider workplace type: remote”
From employer posting metadata - Geographic scope
- PL
“Provider location: Poland”
From employer posting metadata - Employment type
- Temporary
8 fields not stated in this posting
- Employer sponsorship
- Not specified
- Degree requirement
- Not specified
- Security clearance
- Not mentioned
- Timezone / hours
- Not specified
- Citizenship requirement
- Not mentioned
- Minimum experience (years)
- Not specified
- Work authorization
- Not specified
- Compensation
- Not specified
These details were not stated in the tracked posting. Confirm with the employer.
Original job description
Reading view uses normalized posting text so exact evidence excerpts can be highlighted.
Show less description
Original formatting and links
About Samsung Food (Whisk)
At Samsung Food (you might know us as Whisk), we are pioneering the future of everyday care and health — helping individuals and families around the world live better through connected, proactive, and personalized experiences. Our mission is to connect food, health, and home across the Samsung ecosystem, turning data from your devices into personalized guidance that supports you and your loved ones.
We work at the intersection of AI, nutrition, behavior change, and digital health. The Samsung Food app was included in Google Play’s Best of 2020 Everyday Essentials list, has been regularly featured on the Apple App Store, and was nominated for a 2021 Webby Award. We’ve also launched Vision AI, which recognises food and meals using our global Food Genome, and developed deep integrations with Samsung Health and SmartThings.
We are a remote-first team of more than 100 people in over 30 countries, united by our commitment to innovation and impact. As part of Samsung Electronics, we combine the agility of a startup with the scale of the world’s largest consumer electronics company. Join us and help build technology that improves daily life for tens of millions of people across phones, wearables, TVs, and kitchen appliances in the years to come.
Learn more about how we’ve shaped a high-performing global team over the past 14 years.
Summary
Overview
We're building an AI coaching experience inside Samsung Health. Every piece of coaching a user sees is generated, not written. Our platform partner owns the evaluation runtime and functional evals (did the pipeline execute, did the tool call succeed). Nobody owns the layer above that: is the output actually good — right framing, right tone, right structure, does it deliver the coaching logic, does it feel like it's talking to me.
This engagement exists to build that layer: author the failure taxonomy, codify it into a binary rubric, validate an LLM judge against your own human grades, and hand the whole thing to an internal owner. We have a v0.1 rubric and golden scenarios from our product lead, and a functional rubric owned by engineering. We don't have the experience rubric — that's what this engagement produces.
Deliverables by Phase (Overall project ~13 weeks)
1. Failure taxonomy and golden set (within 4 weeks from the contract start date)
At least 150 traces hand-graded with open-coded notes. At least 40 of them come from sparse-data synthetic profiles. The taxonomy has 5–10 categories, each with a frequency count, and the top three failures are shown with their share of all failures. Saturation is evidenced: the last 20 traces produced no new category. At least 50 traces are frozen as a holdout. Head of Product signs off the taxonomy.
2. Rubric v1 and calibration (within 4 weeks from phase 1 completion)
One binary pass/fail criterion per taxonomy category, each with a definition, a pass example and a fail example. Written annotation guidelines. At least 30 traces are double-coded with a second grader, and every disagreement is logged and resolved in writing. A guideline revision log records each change and its reason. Head of Product signs off the rubric.
3. Validated judge and weekly readout (within 4 weeks from phase 2 completion)
Judge prompts for every criterion. A validation report giving true positive rate and true negative rate per criterion on both the dev set and the holdout. A weekly readout on helpfulness, relevance and tone, run at least twice: once by the consultant, and once by the internal owner with the consultant shadowing. A documented revalidation routine, triggered by every model or prompt change and run quarterly regardless.
4. Playbook and handover test (within 1 week from phase 3 completion)
A written playbook covering the whole method: grading, updating the taxonomy, revising the rubric, revalidating the judge and running the readout. The handover test passes when the internal owner reruns the judge validation unaided, gets within ±5 points, and runs one weekly readout with no consultant involvement.
What This Role Is Not
- Not building eval infrastructure, harnesses, or dashboards (owned elsewhere)
- Not functional/module-level evals or accuracy metrics
- Not clinical safety validation
- Not high-volume annotation — grading a few hundred traces to build the taxonomy; volume work goes to vendors/the judge once the rubric exists
What We're Looking For
You've run this loop end-to-end at least once on conversational or generated-text output: error analysis on real traces → a failure taxonomy you built yourself → binary criteria and annotation guidelines → an LLM judge validated against your own labels.
Background: conversation design, AI product quality, model behaviour/policy, content design, human data operations, UX research with strong qualitative coding, or RLHF. Applied linguistics and product management backgrounds also fit.
You'll be strong in this role if you:
- Can describe a failure mode you personally discovered by reading output — one nobody had named before you
- Are comfortable being the arbiter and documenting what you overruled and why
- Understand the rubric is discovered through grading, not written upfront
- Can explain why a judge that agrees with you 94% of the time might still be useless
- Write annotation guidelines that don't need interpreting
- Are comfortable working in notebooks and spreadsheets (no production code required)
Not required: domain expertise in nutrition, weight management, or behaviour change — we have that in-house.
No similar roles found yet — browse all whisk jobs.
