Agent Evaluation Engineer — Build-Time Framework & Deployment Gates

EPAM Systems Inc; United States; Remote; USD 140,000 - 190,000; Full time

Job summary

EPAM Systems is seeking an Agent Evaluation Engineer to design and maintain an evaluation framework for AI agents, including automated tests and CI/CD deployment gates. The role focuses on creating multi-layer evaluation suites that blend deterministic checks with LLM-powered graders, simulating multi-turn conversations, and defining reliability metrics.

You will implement staging validations, shadow-mode traffic analysis, and A/B rollout strategies, with feedback loops from production

Qualifications

• 4+ years building automated testing or evaluation frameworks for ML/LLM/agentic systems.
• Experience designing multi-layer evaluation suites with deterministic and LLM-based graders.
• Expertise with CI/CD pipelines and metric-based quality gates for automated deployments.
• Familiarity with LangGraph or similar agent orchestration frameworks.
• Experience with shadow-mode, canary, or A/B deployment practices for ML platforms.

Responsibilities

• Design and implement build-time evaluation frameworks for agentic workflows using LangGraph or similar orchestration.
• Create deterministic and LLM-as-judge grading pipelines covering reasoning, trajectory accuracy, and output quality.
• Develop test harnesses for multi-turn conversational simulations and context-retention scoring.
• Define reliability assessment methods including multi-trial metrics (pass@k, pass^k).
• Implement CI/CD deployment gates that enforce quality thresholds and block releases not meeting standards.
• Integrate staging validation, shadow-mode traffic comparison, and A/B rollout control in deployment pipelines.
• Leverage AWS AgentCore Evaluations API for on-demand and online scoring components connected to production feedback.
• Convert production incidents into reusable regression cases for continuous quality improvement.
• Collaborate with engineering and DevOps teams to embed evaluation gates into automated workflows.

Skills

Automated testing frameworks

Evaluation frameworks

CI/CD pipelines

LangGraph orchestration

LLM-based graders

AWS AgentCore Evaluations

Shadow-mode/canary/A/B deployment

Multi-turn conversation simulations

Tools

LangGraph

AWS AgentCore Evaluations API

Job description

We're looking for an Agent Evaluation Engineer — Build-Time Framework & Deployment Gates to join our team in Portugal in a fully remote working mode. In this role, you will design and maintain an evaluation framework for AI agents, ensuring quality and compliance through automated tests and CI/CD deployment gates. You will develop multi-layer evaluation suites that blend deterministic checks with LLM-powered graders, simulate multi-turn conversations, and define reliability metrics. The position also involves implementing staging validations, shadow-mode traffic analysis, and A/B rollout strategies, with feedback loops from production environments to enhance overall system robustness.

Responsibilities

• Design and implement build-time evaluation frameworks for agentic workflows using LangGraph or comparable orchestration frameworks
• Create deterministic and LLM-as-judge grading pipelines covering reasoning, trajectory accuracy, and output quality
• Develop test harnesses for multi-turn conversational simulations and context-retention scoring
• Define reliability assessment methods including multi-trial metrics (pass@k, pass^k)
• Implement CI/CD deployment gates that enforce quality thresholds and block releases not meeting standards
• Integrate staging validation, shadow-mode traffic comparison, and A/B rollout control in deployment pipelines
• Leverage AWS AgentCore Evaluations for on-demand and online scoring components connected to production feedback
• Convert production incidents into reusable regression cases for continuous quality improvement
• Collaborate with engineering and DevOps teams to embed evaluation gates into automated workflows

Requirements

• 4+ years of experience in building automated testing or evaluation frameworks for ML, LLM, or agentic systems
• Proven hands‑on experience designing multi-layer evaluation suites with deterministic and LLM-based graders
• Expertise with CI/CD pipelines and implementing metric-based quality gates for automated deployments
• Practical knowledge of LangGraph or similar agent orchestration frameworks
• Strong background in designing simulation-based evaluation strategies and conversation-level tests
• Nice to have Experience with AWS AgentCore Evaluations API (CreateEvaluation, custom evaluators)
• Familiarity with shadow-mode, canary, or A/B deployment practices for ML-based platforms
• Background in transforming production failures into build-time regression tests for agent workflows