Remote Agent Evaluation Engineer - Build-Time Framework

EPAM Systems Inc; Portugal; Remote; EUR 70,000 - 110,000; Full time

Benefits offered by this job

Flexible work hours

Job summary

EPAM Systems Inc in Portugal is seeking an Agent Evaluation Engineer to design and maintain an evaluation framework for AI agents, blending deterministic tests with LLM-powered graders. The role focuses on build-time frameworks, CI/CD gates, and deployment validations within a remote setup.

You will collaborate with SRE and ML teams to implement staging validations, shadow-mode traffic analysis, and A/B rollout strategies, driving reliability and quality across production environments.

Qualifications

• 4+ years of experience building automated testing or evaluation frameworks for ML/LLM/agentic systems
• Experience designing multi-layer evaluation suites with deterministic and LLM-based graders
• Expertise with CI/CD pipelines and implementing metric-based quality gates for automated deployments
• Practical knowledge of LangGraph or similar agent orchestration frameworks
• Strong background in simulation-based evaluation strategies and conversation-level tests

Responsibilities

• Design and implement build-time evaluation frameworks for agentic workflows using LangGraph or comparable orchestration frameworks
• Create deterministic and LLM-as-judge grading pipelines covering reasoning, trajectory accuracy, and output quality
• Develop test harnesses for multi-turn conversational simulations and context-retention scoring
• Define reliability assessment methods including multi-trial metrics (pass@k, pass^k)
• Implement CI/CD deployment gates that enforce quality thresholds and block releases not meeting standards
• Integrate staging validation, shadow-mode traffic comparison, and A/B rollout control in deployment pipelines
• Leverage AWS AgentCore Evaluations for on-demand and online scoring components connected to production feedback
• Convert production incidents into reusable regression cases for continuous quality improvement
• Collaborate with engineering and DevOps teams to embed evaluation gates into automated workflows

Job description

EPAM Systems Inc in Portugal is seeking an Agent Evaluation Engineer to design and maintain an evaluation framework for AI agents, blending deterministic tests with LLM-powered graders. The role focuses on build-time frameworks, CI/CD gates, and deployment validations within a remote setup.

You will collaborate with SRE and ML teams to implement staging validations, shadow-mode traffic analysis, and A/B rollout strategies, driving reliability and quality across production environments.