Senior ML Evaluation Engineer (Remote) — CI/CD QA

EPAM Systems, Inc.; Portugal; Teletrabalho; EUR 70 000 - 110 000; Tempo integral

Resumo da oferta

EPAM Systems, Inc. is seeking a Senior ML / Evaluation Engineer to design and implement advanced evaluation frameworks for an Enterprise Agent Development Platform. You will build LLM-as-judge evaluators and deterministic validators to enforce quality before release.

You will collaborate with platform, orchestration, and DevOps teams to ensure scalable, observable evaluation in production, leveraging AWS services, OpenTelemetry, and CloudWatch for monitoring and scoring.

Qualificações

• 5+ years of ML engineering or AI evaluation experience.
• Hands-on experience designing LLM evaluation frameworks.
• Experience implementing CI/CD gates for ML QA.
• Proficiency in Python for evaluation logic.

Responsabilidades

• Design and implement multi-layer evaluation frameworks for agent workflows.
• Build LLM-as-judge evaluators using AWS AgentCore modules.
• Develop deterministic Lambda evaluators for rule-based checks.
• Define enterprise evaluation standards with scoring criteria.
• Integrate CI/CD deployment gates for automated pipelines.
• Enable online evaluation in production with sampling strategies.

Conhecimentos

ML engineering

AI evaluation frameworks

LLM evaluation

CI/CD deployment

Python

OpenTelemetry

Observability

Cloud monitoring

Ferramentas

AWS AgentCore

AWS Bedrock

AWS Lambda

CloudWatch

OpenTelemetry

Descrição da oferta de emprego

EPAM Systems, Inc. is seeking a Senior ML / Evaluation Engineer to design and implement advanced evaluation frameworks for an Enterprise Agent Development Platform. You will build LLM-as-judge evaluators and deterministic validators to enforce quality before release.

You will collaborate with platform, orchestration, and DevOps teams to ensure scalable, observable evaluation in production, leveraging AWS services, OpenTelemetry, and CloudWatch for monitoring and scoring.