Senior AI Evaluations Engineer — AWS, Remote

TechnoSphere, Inc.; United States; A distancia; USD 180.000 - 240.000; Jornada completa

Descripción de la vacante

TechnoSphere, Inc. is seeking a Senior AI Evaluations Engineer to join AWS ProServe remotely. You will design and operate evaluation frameworks ensuring agentic AI systems are accurate, safe, and production-ready for large enterprise customers, particularly in financial services.

The role requires expert GenAI engineering, rubric design, and building scalable evaluation pipelines using Bedrock, Guardrails, CloudWatch, Lambda, and S3. AWS certifications are required.

Formación

• Expert-level GenAI experience with advanced Python.
• Experience building and running production-grade systems.
• Hands-on agent evaluation including offline/online testing.
• Design and scoring at session, response and tool-call granularity.
• Experience with evaluation metrics and safety considerations.
• Hands-on with Strands, LangGraph, CrewAI or similar.
• Familiar with Amazon Bedrock, Guardrails, CloudWatch, Lambda, S3, IAM.
• Active AWS Certified Generative AI Developer – Professional or AWS ML Engineer – Associate.

Responsabilidades

• Design and operate evaluation frameworks for agentic AI systems.
• Define quality rubrics and scoring architectures for production use.
• Build LLM-as-judge and agent-as-judge pipelines.
• Run offline regression and online live-traffic evaluation for multi-agent workflows on Bedrock.

Conocimientos

GenAI Expertise

Production Systems

Agent Evaluation

LLM-as-Judge

Evaluation Metrics

Agentic Frameworks

AWS Stack

AWS Certification

Descripción del empleo

TechnoSphere, Inc. is seeking a Senior AI Evaluations Engineer to join AWS ProServe remotely. You will design and operate evaluation frameworks ensuring agentic AI systems are accurate, safe, and production-ready for large enterprise customers, particularly in financial services.

The role requires expert GenAI engineering, rubric design, and building scalable evaluation pipelines using Bedrock, Guardrails, CloudWatch, Lambda, and S3. AWS certifications are required.