Full-Time (GC/USC only) Role: Senior AI Evaluations Engineer (AWS)_ Remote

TechnoSphere, Inc.; United States; Remote; USD 180,000 - 240,000; Full time

Job summary

TechnoSphere, Inc. is seeking a Senior AI Evaluations Engineer to join AWS ProServe remotely. You will design and operate evaluation frameworks ensuring agentic AI systems are accurate, safe, and production-ready for large enterprise customers, particularly in financial services.

The role requires expert GenAI engineering, rubric design, and building scalable evaluation pipelines using Bedrock, Guardrails, CloudWatch, Lambda, and S3. AWS certifications are required.

Qualifications

• Expert-level GenAI experience with advanced Python.
• Experience building and running production-grade systems.
• Hands-on agent evaluation including offline/online testing.
• Design and scoring at session, response and tool-call granularity.
• Experience with evaluation metrics and safety considerations.
• Hands-on with Strands, LangGraph, CrewAI or similar.
• Familiar with Amazon Bedrock, Guardrails, CloudWatch, Lambda, S3, IAM.
• Active AWS Certified Generative AI Developer – Professional or AWS ML Engineer – Associate.

Responsibilities

• Design and operate evaluation frameworks for agentic AI systems.
• Define quality rubrics and scoring architectures for production use.
• Build LLM-as-judge and agent-as-judge pipelines.
• Run offline regression and online live-traffic evaluation for multi-agent workflows on Bedrock.

Skills

GenAI Expertise

Production Systems

Agent Evaluation

LLM-as-Judge

Evaluation Metrics

Agentic Frameworks

AWS Stack

AWS Certification

Job description

Job Title: Senior AI Evaluations Engineer (AWS)

Location: Remote

Job Type: Remote

We are seeking a Senior AI Evaluations Engineer to AWS practice and work embedded with AWS Professional Services (ProServe) teams, designing and operating the evaluation frameworks that determine whether agentic AI systems are accurate, safe and ready for production at large enterprise customers, primarily in financial services. You will define quality rubrics and scoring architectures, build LLM-as-judge and agent-as-judge pipelines, and run offline regression and online, live-traffic evaluation for multi-agent workflows on Amazon Bedrock. The ideal candidate combines expert generative AI engineering skills with a rigorous, metrics-driven approach to quality.

Required Qualifications

• GenAI Expertise: Expert-level (Level 400) generative AI experience with advanced Python.
• Production Systems: Proven experience building and running production-grade systems.
• Agent Evaluation: Hands-on experience with offline/on-demand regression testing and online/continuous live-traffic scoring of agents.
• LLM-as-Judge: Proficiency in rubric design and scoring at session, response and tool-call granularity.
• Evaluation Metrics: Task completion, tool-call accuracy, trajectory quality, groundedness and safety.
• Agentic Frameworks: Hands-on experience with Strands, LangGraph, CrewAI or similar.
• AWS Stack: Amazon Bedrock (models, Guardrails) and CloudWatch, plus Lambda, S3 and IAM fundamentals.
• AWS Certification: Active AWS Certified Generative AI Developer – Professional or AWS Certified Machine Learning Engineer – Associate (required).

Preferred Qualifications

• Bedrock AgentCore Evaluations: On-demand SDK runner, online monitoring and A/B testing.
• Advanced Evaluation Methods: Agent-as-a-judge frameworks and human-in-the-loop (HITL) evaluation pipelines.
• Benchmarking: Benchmarking frameworks (tau-bench, SWE-bench, GAIA) and custom dataset construction.
• RAG Evaluation: RAG quality metrics such as faithfulness, context precision and recall.
• CI/CD & Observability: CI/CD integration for evaluation regression, tracing and structured logging.
• Legacy Modernization: Familiarity with Java/Spring, Sybase, DB2 or COBOL as source context for agentic modernization or integration pattern design.
• Financial Services: Financial services or regulated-industry experience with compliance-aware delivery (trade surveillance, data controls, audit requirements).
• Scale: Large-scale portfolio delivery: reusable patterns or components applied across 50+ applications, including wave planning and throughput measurement.
• Consulting: Prior experience with AWS ProServe or in a consulting delivery role.