Hiring: AI Engineer | Agentic AI, AWS Bedrock AgentCore & Langfuse | 15+ Years

TMS LLC; Los Angeles (CA); On-site; USD 120,000 - 180,000; Full time

Job summary

Agentic AI is seeking a hands-on AI Engineer to build, deploy, monitor, and continuously improve production-grade AI agents using Python, Agentic AI, AWS Bedrock AgentCore, and Langfuse.

The role focuses on developing agents, deploying on Bedrock, designing golden datasets, and implementing LLM-as-a-Judge evaluations with CI/CD gates and observability tooling such as Langfuse/OpenTelemetry.

Qualifications

• Hands-on Python development experience in AI agents.
• Experience with Agentic AI and AWS Bedrock AgentCore evaluations.
• Design and curating golden datasets for model evaluation.
• Implementing automated evaluation and CI gate integration.
• Experience with Langfuse/OpenTelemetry for observability.
• Familiarity with RAG frameworks and agentic testing.

Responsibilities

• Develop AI agents using Python and Agentic AI.
• Deploy agents on AWS Bedrock AgentCore Runtime.
• Design golden datasets and conduct LLM evaluations.
• Integrate automated evaluations into CI/CD pipelines as quality gates.
• Implement Langfuse/OpenTelemetry for observability and tracing.
• Evaluate agent accuracy, task completion, tool usage, and performance.
• Support RAG pipelines and related data workflows.

Skills

Python

Agentic AI development

LLM evaluation

CI/CD

Troubleshooting

Problem solving

Tools

AWS Bedrock AgentCore

Langfuse

OpenTelemetry

GitHub Actions

Job description

Hiring: AI Engineer | Agentic AI, AWS Bedrock AgentCore & Langfuse | 15+ Years

• Contract

Job Summary

We are seeking a hands‑on AI Engineer with expertise in Python, Agentic AI, AWS Bedrock AgentCore, and LLM evaluation to build, deploy, monitor, and continuously improve production‑grade AI agents.

Key Responsibilities

Develop AI agents using Python and Claude Agent SDK.

Deploy agents using AWS Bedrock AgentCore Runtime.

Design golden datasets and implement LLM-as-a-Judge evaluations.

Integrate automated evaluations into CI/CD pipelines as quality gates.

Implement Langfuse/OpenTelemetry for AI observability and tracing.

Evaluate agent accuracy, task completion, tool execution, and performance.

Support RAG pipelines and AWS data processing workflows.

Required Skills

Python and Agentic AI development.

AWS Bedrock AgentCore Evaluations / Bedrock evaluation.

Golden-dataset design and curation.

LLM-as-a-Judge design and calibration against human labels.

Automated evaluation and CI gate integration.

Langfuse and/or OpenTelemetry.

Good to Have

RAGAS or equivalent evaluation frameworks.

Trajectory evaluation and agentic red‑teaming.

Bedrock Guardrails, Strands Agents SDK, and GitHub Actions.

Qualifications

6-8+ years of software engineering, cloud engineering, or AI/ML experience.

Hands‑on experience with production AI agents and continuous evaluation.

Strong problem‑solving and troubleshooting skills.

All your information will be kept confidential according to EEO guidelines.