AI Engineer / Generative AI Engineer

TMS LLC; Los Angeles (CA); Hybrid; USD 180,000 - 280,000; Full time

Job summary

TMS LLC is seeking an AI Engineer / Generative AI Engineer to help build production-grade AI agents and bring them through deployment, monitoring, and continuous improvement. This contract role is based in Los Angeles, CA with a hybrid work model, and focuses on agent development, evaluation, and observability in real environments.

You will work with Python and the Claude Agent SDK, deploy via AWS Bedrock AgentCore, and strengthen quality through golden datasets, LLM-as-a-Judge evaluations, and

Qualifications

• 15+ years of experience is required.
• 6–8+ years of software engineering, cloud engineering, or AI/ML experience.
• Hands-on experience with production AI agents and continuous evaluation.
• Strong problem-solving and troubleshooting skills.

Responsibilities

• Develop AI agents using Python and Claude Agent SDK.
• Deploy agents via AWS Bedrock AgentCore Runtime.
• Design and maintain golden datasets for reliable evaluation.
• Implement LLM-as-a-Judge evaluations and calibrate against human labels.
• Integrate automated evaluation into CI/CD quality gates.
• Set up Langfuse and/or OpenTelemetry for observability, tracing, and monitoring.
• Evaluate agent outcomes including accuracy, task completion, tool execution, and performance.
• Support RAG pipelines and AWS data processing workflows.

Skills

Python

agentic AI development

LLM evaluation design

CI/CD quality gates

Troubleshooting

Tools

Claude Agent SDK

AWS Bedrock AgentCore

Langfuse

OpenTelemetry

Golden datasets

LLM-as-a-Judge

Job description

TMS LLC is seeking an AI Engineer / Generative AI Engineer to help build production-grade AI agents and bring them through deployment, monitoring, and continuous improvement. This contract role is based in Los Angeles, CA with a hybrid work model, and focuses on agent development, evaluation, and observability in real environments.

The work centers on developing and operating AI agents using Python and the Claude Agent SDK, deploying via AWS Bedrock AgentCore, and strengthening quality through golden datasets, LLM-as-a-Judge evaluations, and CI/CD quality gates.

What you will do

• Develop AI agents using Python and Claude Agent SDK.
• Deploy agents using AWS Bedrock AgentCore Runtime.
• Design and maintain golden datasets for reliable evaluation coverage.
• Implement LLM-as-a-Judge evaluations, including calibration against human labels.
• Integrate automated evaluation workflows into CI/CD pipelines as quality gates.
• Set up Langfuse and/or OpenTelemetry for AI observability, tracing, and monitoring.
• Evaluate agent outcomes including accuracy, task completion, tool execution, and performance.
• Support RAG pipelines and AWS data processing workflows.

Required skills

• Experience in Python and agentic AI development.
• Hands‑ on work with AWS Bedrock AgentCore Evaluations (including Bedrock evaluation).
• Ability to create, curate, and use golden datasets.
• Experience designing LLM-as-a-Judge systems and calibrating them against human labels.
• Automated evaluation implementation and integration into CI gates.
• Experience with Langfuse and/or OpenTelemetry.

Technology stack

• Python, Claude Agent SDK
• AWS Bedrock AgentCore, AWS Bedrock AgentCore Runtime
• LLM-as-a-Judge, CI/CD
• Langfuse, OpenTelemetry
• RAG, AWS data processing workflows
• Bedrock evaluation

Nice to have

• RAGAS or equivalent evaluation frameworks.
• Trajectory evaluation and agentic red‑teaming.
• Bedrock Guardrails, Strands Agents SDK, and GitHub Actions.

Qualifications

• 15+ years of experience (minimum).
• 6–8+ years of software engineering, cloud engineering, or AI/ML experience.
• Hands‑on experience with production AI agents and continuous evaluation.
• Strong problem‑solving and troubleshooting skills.