Applied AI Researcher: Model Evaluation & Data Strategy

Verita AI; San Francisco (CA); On-site; USD 120,000 - 180,000; Full time

Job summary

Verita AI is seeking an Applied AI Researcher to work with clients on model evaluation and data strategy. You will assess model performance, identify failure modes, and design data-driven solutions, collaborating with operations and engineering to implement scalable data programs.

The role requires strong Python skills, experience with model APIs, and the ability to communicate findings to clients and stakeholders clearly.

Qualifications

• Experience in applied AI research, machine learning, model evaluation, or data-centric AI.
• Experience evaluating foundation models or generative AI systems.
• Strong Python skills and experience working with model APIs and structured datasets.
• Ability to translate model failures into practical data solutions.
• Strong technical writing and client communication skills.

Responsibilities

• Work with clients to understand their models, goals, and performance gaps.
• Design evaluations for generative, multimodal, reasoning, tool-use, and agentic AI systems.
• Analyze model outputs and benchmark results to identify and quantify failure modes.
• Recommend data solutions such as supervised fine-tuning data, preference data, expert demonstrations, critiques, and evaluation datasets.
• Write client proposals covering the methodology, data design, quality controls, staffing, deliverables, and expected impact.
• Create annotation guidelines, scoring rubrics, gold-standard tasks, and evaluator-training programs.
• Design pilot studies and measure whether data interventions improve model performance.
• Build quality systems using calibration tasks, blind review, adjudication, and expert scoring.
• Work with operations and engineering teams to launch and scale data pipelines.
• Present findings and recommendations to clients.

Job description

Verita AI is seeking an Applied AI Researcher to work with clients on model evaluation and data strategy. You will assess model performance, identify failure modes, and design data-driven solutions, collaborating with operations and engineering to implement scalable data programs.

The role requires strong Python skills, experience with model APIs, and the ability to communicate findings to clients and stakeholders clearly.