[Remote] Evaluation Lead, AI Operations

Note: The job is a remote job and is open to candidates in USA. Fetch is a company focused on transforming everyday activities into meaningful rewards. They are seeking an Evaluation Lead for AI Operations to own automation and evaluation programs, ensuring quality and measurable impact of their AI systems.

Responsibilities

• Own programs: Lead complex, high-impact automation and evaluation initiatives across workflows or teams
• Design scalable solutions: Architect end-to-end workflows integrating datasets, evaluations, automations, and HITL processes
• Establish standards: Define best practices for datasets, evaluation methodologies, and quality measurement
• Drive cross-functional alignment: Influence priorities and partner with Engineering, Product, and AI teams to deliver solutions
• Improve systems at scale: Lead redesign of workflows, tooling, and processes to improve performance and durability
• Measure and communicate impact: Define success metrics and communicate performance and recommendations to leadership
• Elevate the team: Raise the bar through your work and actively mentor others by sharing approaches, guiding problem-solving, and enabling the team to build stronger automation and evaluation capabilities
• Drive innovation: Stay current on emerging tools and approaches; pilot and translate them into actionable improvements for the team

Skills

• 5+ years of experience and the proven ability to lead complex automation or evaluation initiatives across systems or teams
• Experience designing, building, and scaling evaluation frameworks, datasets, and quality systems at scale for LLMs or AI products
• Strong understanding of AI/LLM workflows, automation platforms, and system design
• Experience in influencing cross-functional stakeholders and aligning priorities
• Demonstrated ability to define metrics and drive measurable business impact
• Familiarity with data pipelines, APIs, or production systems
• Experience mentoring or leading technical contributors
• Background in AI Operations, applied AI, or process optimization

Benefits

• Equity: We offer full-time employees equity in Fetch, so that everyone can benefit from Fetch’s growth.
• 401k Match: Dollar-for-dollar match up to 4%.
• Benefits for humans and pets: We offer comprehensive medical, dental and vision plans for everyone including your pets.
• Continuing Education: Fetch provides ten thousand per year in education reimbursement.
• Employee Resource Groups: Take part in employee-led groups that are centered around fostering a diverse and inclusive workplace through events, dialogue and advocacy. The ERGs participate in our Inclusion Council with members of executive leadership.
• Paid Time Off: On top of our flexible PTO, Fetch observes 9 paid holidays, as well as our year-end week-long break.
• Robust Leave Policies: 20 weeks of paid parental leave for primary caregivers, 14 weeks for secondary caregivers, and a flexible return to work schedule.
• Calvin Care Cash: Employees who are welcoming new family members will also receive a one time $2,000 incentive to assist employees with covering the cost of childcare, clothing, diapers and much more!
• Flexible Work Environment: Collaborate with your team in one of our stunning offices, or you can work fully remotely from anywhere in the US. We’ll ensure you are equally equipped with the hardware and software you need to get your job done in the comfort of your home. (applicable for most roles)

Company Overview

Company H1B Sponsorship