AI Engineer, Evaluation

Distyl AI; New York (NY); Hybrid; USD 150,000 - 250,000; Full time

Benefits offered by this job

Equity

Medical insurance

Flexible time off

401(k)

Job summary

Distyl AI in New York, NY, hybrid, is seeking an AI Engineer, Evaluation to design and deploy evaluation frameworks for AI systems in customer environments.

You will specify quality metrics, build golden test cases and regression suites in Python, and create offline/online evaluation pipelines to inform model design and release readiness. Travel 10-50% may be required.

Qualifications

• 2+ years of software engineering experience.
• Strong Python engineering skills to build production-grade evaluation pipelines.
• Experience with Evaluation-Driven or Experiment-Driven development.
• Ability to translate human judgment into test cases, scoring functions and graders.
• Systems-oriented mindset for prompts, agents, data and deployment interactions.
• Willingness to travel 10-50%.

Responsibilities

• Craft and deploy evaluation frameworks for AI systems in customer environments.
• Specify domain-specific quality metrics reflecting user needs and business goals.
• Create and maintain golden test cases and regression suites in Python.
• Build offline and online evaluation pipelines integrated into iteration loops.
• Inform prompt design, agent logic, model selection, and release readiness using evaluation results.
• Define and operate LLM-based graders aligned with expert assessments.
• Investigate divergences between evaluation signals and real-world outcomes.
• Collaborate with engineers, architects, product teams and domain experts.

Skills

Python

Software engineering

Evaluation-driven development

Test case design

Collaboration

Travel readiness

Tools

Python tooling

Job description

Distyl AI is seeking an AI Engineer, Evaluation in New York, NY (hybrid) with a base salary of USD 150,000-250,000 per year.

Responsibilities

• Craft and deploy evaluation frameworks that support Evaluation-Driven Development for AI systems operating in customer environments
• Specify domain specific quality metrics so evaluation signals reflect real user needs, constraints, and business goals
• Create and maintain golden test cases and regression suites in Python, using both human-authored and AI-assisted test generation to capture key behaviors and edge cases
• Treat these test suites as core system components that evolve alongside the AI system
• Build and sustain offline and online evaluation pipelines that integrate into system iteration loops
• Use evaluation results to inform prompt design, agent logic, model selection, and release readiness to drive measurable improvements
• Define, calibrate, and operate LLM-based graders, aligning automated judgments with expert human assessments
• Investigate divergences between evaluation signals and real-world outcomes and refine grading approaches as systems and domains evolve
• Collaborate with Forward Deployed AI Engineers, Architects, Product Engineers, AI Strategists, and domain experts to ensure evaluation frameworks guide production development and deployment

Requirements

• 2+ years of software engineering experience
• Strong Python engineering skills: write clean, maintainable Python and build production-grade evaluation and experimentation pipelines
• Experience with Evaluation-Driven or Experiment-Driven Development: familiarity with structured evaluation or experimentation frameworks to drive iteration and awareness of overfitting risks
• Ability to translate human judgment into code: collaborate with subject matter experts to encode high-quality judgments into test cases, scoring functions, and scalable graders
• Systems-oriented mindset: understand how evaluation interacts with prompts, agents, data, and deployment; enable fast iteration while preserving trust and safety in production
• AI-native working style: leverage AI tools to generate tests, analyze failures, explore edge cases, and accelerate debugging
• Travel: 10-50% of the time, depending on project and role

Technologies

• Python

Benefits

• Base salary range of USD 150K-250K plus meaningful equity and comprehensive benefits
• 100% coverage of medical, dental, and vision insurance for employees and their dependents
• Flexible time off policy
• Retirement and financial planning benefits, including pre-tax HSA, FSA, commuter accounts, 401(k), and financial coaching
• Wellness benefits covering physical and mental health, plus fertility and family-building support via Carrot
• Complimentary in-office lunches and snacks
• Access to state-of-the-art AI models and tools to tackle real-world business problems
• Opportunity to own high-impact projects across major enterprises
• Fast-paced, mission-driven culture that values curiosity, pragmatism, and excellence