Python Engineer - AI Evaluator Library

Intellias; Portugal; Presencial; EUR 55 000 - 75 000; Tempo integral

Vantagens oferecidas por esta oferta de emprego

Annual leave 22 days

Sick leave 3 days

Permanent contract

Health insurance

Office Porto

Resumo da oferta

Intellias is seeking a Python Engineer - Evaluator Library in Porto to design and implement reusable evaluation components for enterprise AI agents and LLM-powered workflows. You will build custom evaluation capabilities focused on automated quality validation, workflow compliance, PII protection, and structured output verification, collaborating with AI Platform, ML, and DevOps teams.

The role emphasizes creating deterministic quality checks, integrating with AWS Lambda and Bedrock, and

Qualificações

• 4+ years Python engineering.
• LLM evaluation or quality assurance for AI/ML systems.
• AWS Lambda function development and deployment.

Responsabilidades

• Design, develop, and maintain reusable Python-based evaluator libraries for AI agents and LLM-powered workflows.
• Implement AWS Lambda-based custom evaluators to perform deterministic quality, compliance, and validation checks.
• Develop LLM-as-a-judge evaluation logic to assess subjective dimensions such as relevance, helpfulness, consistency, and response quality.
• Build automated PII detection evaluators using regex-based techniques and AWS Bedrock Guardrails integrations.
• Implement TOOL_CALL-level validation mechanisms to verify structured outputs, JSON schema compliance, and tool response correctness.
• Develop SESSION-level evaluators to validate workflow contract compliance, execution integrity, and cross-step behavioral expectations.
• Create TRACE-level evaluators for numerical accuracy verification, calculation consistency, and deterministic result validation.
• Integrate evaluator components with AWS AgentCore Evaluation workflows and enterprise AI quality pipelines.
• Collaborate with AI platform teams to define evaluation standards, scoring methodologies, and quality acceptance criteria.
• Design and maintain unit, integration, and validation tests for evaluator libraries and quality frameworks.
• Support observability and troubleshooting by integrating evaluation outputs with CloudWatch logging and monitoring capabilities.

Conhecimentos

Python

LLM evaluation

AWS Lambda

PII detection

Quality assurance

Regex

AWS Bedrock

CloudWatch

Evaluation standards

Ferramentas

AWS Lambda

AWS Bedrock

Regex

CloudWatch

Descrição da oferta de emprego

Intellias is seeking a Python Engineer - Evaluator Library in Porto to design and implement reusable evaluation components for enterprise AI agents and LLM-powered workflows. You will build custom evaluation capabilities focused on automated quality validation, workflow compliance, PII protection, and structured output verification, collaborating with AI Platform, ML, and DevOps teams.

The role emphasizes creating deterministic quality checks, integrating with AWS Lambda and Bedrock, and