LLM - AI Quality Analyst (Personalization)

Opening for 350 AI Quality Analysts to evaluate cutting-edge personalization features for large language models. In this role, you will design real-world conversational prompts and evaluate how accurately, naturally, and safely an AI incorporates personal contextual data into its responses.

Key Responsibilities

• Design and execute 1–5 turn conversational prompts rooted in your personal context and experiences.
• Conduct Side-by-Side (SxS) model response evaluations across dimensions such as Grounding, Integration, and Helpfulness.
• Identify personalization errors, unsupported inferences, hallucinations, and robotic over-narration.
• Write clear, structured, and defensible rationales explaining model rankings with turn-specific references.
• Inspect model debug data to confirm appropriate context retrieval while maintaining strict session data hygiene.

Basic Requirements

• Experience: 1+ years of professional experience in data annotation, AI quality evaluation, content moderation, or related analytical fields (Linguistics, Law, Journalism, Computer Science, etc.).
• Language Proficiency: Native or bilingual English reading and writing skills.
• Account Requirement: Willingness to use your primary, active personal Google account (Gmail, Search, YouTube history) to test real-world personalization features.
• Availability: Available 30 to 40 hours per week, with at least 4 hours of daily overlap with US Pacific Standard Time (PST).
• Work Setup: Reliable personal laptop/desktop with high-speed internet.

Hiring Process

• Application & Job Interest Form
• Online Evaluation Assessment (Must be completed within 24 hours of receipt)
• Pre-onboarding & Offer

Pay: $30.00 per hour

Expected hours: 30.0 – 40.0 per week

Benefits

* Flexible schedule

Work Location: Remote