Remote AI Evaluation Engineer: Safeguard & Scale

DeepRec.ai; Denver (CO); Remote; USD 162,000 - 198,000; Full time

Job summary

A mission-driven tech company is seeking an AI Evaluation Engineer to design and own evaluation systems that safeguard AI features. In this role, you will create frameworks and tools to ensure that AI is deployed safely and accurately. The ideal candidate has a strong software engineering background and experience with OpenAI API or similar LLM tools. Join a team that empowers professionals to deliver critical support through AI capabilities while prioritizing safety and innovation.

Qualifications

• Experience with TypeScript is a plus.
• Practical knowledge of function calling and LLM grading.
• Ability to validate data quality and performance.

Responsibilities

• Design frameworks for evaluation outputs.
• Build data pipelines and integrate with CI.
• Conduct red team assessments of AI systems.
• Establish model versioning and observability.
• Deliver tooling and dashboards for engineers.

Skills

Strong software engineering background

Deep experience with OpenAI API or similar LLM ecosystems

Practical knowledge of prompting and eval techniques

Familiarity with statistical analysis

Experience with observability or data science tooling

Job description

A mission-driven tech company is seeking an AI Evaluation Engineer to design and own evaluation systems that safeguard AI features. In this role, you will create frameworks and tools to ensure that AI is deployed safely and accurately. The ideal candidate has a strong software engineering background and experience with OpenAI API or similar LLM tools. Join a team that empowers professionals to deliver critical support through AI capabilities while prioritizing safety and innovation.