AI Engineer — Production Agents (AWS Bedrock & Langfuse)

Technology Management Solutions LLC; Los Angeles (CA); On-site; USD 120,000 - 180,000; Full time

Job summary

Agentic AI is seeking a hands-on AI Engineer to build, deploy, monitor, and continuously improve production-grade AI agents using Python, Agentic AI, AWS Bedrock AgentCore, and Langfuse.

The role focuses on developing agents, deploying on Bedrock, designing golden datasets, and implementing LLM-as-a-Judge evaluations with CI/CD gates and observability tooling such as Langfuse/OpenTelemetry.

Qualifications

• Hands-on Python development experience in AI agents.
• Experience with Agentic AI and AWS Bedrock AgentCore evaluations.
• Design and curating golden datasets for model evaluation.
• Implementing automated evaluation and CI gate integration.
• Experience with Langfuse/OpenTelemetry for observability.
• Familiarity with RAG frameworks and agentic testing.

Responsibilities

• Develop AI agents using Python and Agentic AI.
• Deploy agents on AWS Bedrock AgentCore Runtime.
• Design golden datasets and conduct LLM evaluations.
• Integrate automated evaluations into CI/CD pipelines as quality gates.
• Implement Langfuse/OpenTelemetry for observability and tracing.
• Evaluate agent accuracy, task completion, tool usage, and performance.
• Support RAG pipelines and related data workflows.

Skills

Python

Agentic AI development

LLM evaluation

CI/CD

Troubleshooting

Problem solving

Tools

AWS Bedrock AgentCore

Langfuse

OpenTelemetry

GitHub Actions

Job description

Agentic AI is seeking a hands-on AI Engineer to build, deploy, monitor, and continuously improve production-grade AI agents using Python, Agentic AI, AWS Bedrock AgentCore, and Langfuse.

The role focuses on developing agents, deploying on Bedrock, designing golden datasets, and implementing LLM-as-a-Judge evaluations with CI/CD gates and observability tooling such as Langfuse/OpenTelemetry.