Platform Engineer: CI/CD Quality Gates & Online Evaluation

Intellias; Spain (TX); On-site; USD 81,000 - 115,000; Full time

Job summary

Intellias is seeking a Platform Engineer — CI/CD Gate & Online Evaluation to help establish automated quality controls for enterprise AI platforms and agent-based systems. In this role, you will integrate evaluation workflows into deployment pipelines, configure online quality monitoring, and build operational visibility for AI performance and reliability.

You will work closely with platform, DevOps, AI engineering, and governance teams to ensure AI agents meet quality standards before and after

Qualifications

• 4+ years platform or DevOps engineering experience.
• CI/CD pipeline integration for quality gates.
• CloudWatch dashboards and alarm design experience.
• AWS AgentCore Evaluation API or similar.
• GitHub Actions integration for evaluation deployment gate.
• Online evaluation configuration and production sampling knowledge.
• CloudWatch dashboards for per-agent evaluation scores over time.
• CloudWatch alarm configuration for quality degradation thresholds.

Responsibilities

• Design and implement CI/CD quality gates using AWS AgentCore Evaluation to validate AI agents and workflows before deployment.
• Integrate evaluation execution into GitHub Actions and deployment pipelines to support automated release decisions.
• Configure and maintain on-demand evaluation workflows for pre-release quality validation and regression detection.
• Implement and manage online evaluation capabilities, including production sampling strategies and evaluation scheduling.
• Build CloudWatch dashboards that provide visibility into evaluation scores, quality trends, and agent performance over time.
• Configure CloudWatch alarms and notification mechanisms to detect quality degradation and reliability issues.
• Collaborate with AI engineers to define evaluation thresholds, release criteria, and quality acceptance standards.
• Support platform-wide observability by integrating evaluation results into monitoring and reporting workflows.
• Analyze evaluation outcomes and identify quality regressions across agent versions, prompts, tools, and workflows.
• Develop automation enabling continuous validation of AI systems throughout the SDLC.

Skills

DevOps engineering

CI/CD gates

CloudWatch dashboards

AgentCore

GitHub Actions

Online evaluation

Monitoring

Observability

Tools

GitHub Actions

AWS AgentCore

CloudWatch

Job description

Intellias is seeking a Platform Engineer — CI/CD Gate & Online Evaluation to help establish automated quality controls for enterprise AI platforms and agent-based systems. In this role, you will integrate evaluation workflows into deployment pipelines, configure online quality monitoring, and build operational visibility for AI performance and reliability.

You will work closely with platform, DevOps, AI engineering, and governance teams to ensure AI agents meet quality standards before and after