Senior Python AI Engineer - Production LLMs & Agents

Limestone Digital; Central (LA); Remote; USD 160,000 - 220,000; Full time

Job summary

Limestone Digital is seeking an experienced software engineer to lead the development of an AI use case 1 agent within a client environment. The role emphasizes tool integration, retrieval, evaluation, and secure handling of data on both Azure OpenAI and Bedrock platforms.

You will collaborate with client engineers, version control prompts and configurations, and ensure rapid delivery by day 14, with full handover by day 45. Strong Python, CI/CD, and communication skills are essential.

Qualifications

• 5+ years of software engineering, Python, with code review and CI as daily practice.
• 2+ years building production-grade LLM applications: agents with tool calling, structured outputs, RAG.
• Hands-on Azure OpenAI (Microsoft Foundry) or Amazon Bedrock, across clouds.
• Retrieval done properly: chunking, embeddings, hybrid search, reranking, access control enforced in the index.
• Evaluation-first mindset: golden sets and automated evals with the agent, knows LLM-as-judge limits.
• Prompts, configs and agent definitions versioned in Git and released through CI, basic Terraform.
• Careful with confidential client data: least privilege, private networking, PII handling, no data leaves client tenant.
• Explains trade-offs to client architects with evidence, writes decision records and runbooks.
• Pairs with and coaches client engineers; writes runbooks for repeatability.
• English C1 or higher, daily direct work with a US client.
• Based in EU (Poland or Czechia preferred) or Brazil.

Responsibilities

• Build the use case 1 agent on the client's shared services: model calls, tool integration, retrieval and human sign-off steps.
• Get a thin working version running in the client's environment by day 14.
• Set up the agent and prompt registry: Git as system of record, one manifest per agent and prompt, CI checks.
• Build retrieval for use case 1 with access enforced in data; add cross-engagement leak cases to eval set.
• Run every prompt or model change through the evaluation harness and close gaps before client evaluation (days 29–33).
• Write decision records for the design authority and runbooks for operating the agent.
• Pair daily with client engineers; weeks 1–2 together, weeks 3–4 co-work, weeks 5–6 client-led.
• By day 45: client engineers onboard a second use case without us; all code in client repos.

Skills

Python

Code reviews

CI/CD

LLM applications

Azure OpenAI / Bedrock

Cloud-Agnostic (Azure/AWS)

Retrieval & RAG

Evaluation workflows

Prompt engineering & CI versioning

Security & data privacy

English communication (C1+)

Tools

Git

Terraform

CI tooling

Azure OpenAI

Bedrock

OpenSearch/Vector search

Llama/GPT tooling

Job description

Limestone Digital is seeking an experienced software engineer to lead the development of an AI use case 1 agent within a client environment. The role emphasizes tool integration, retrieval, evaluation, and secure handling of data on both Azure OpenAI and Bedrock platforms.

You will collaborate with client engineers, version control prompts and configurations, and ensure rapid delivery by day 14, with full handover by day 45. Strong Python, CI/CD, and communication skills are essential.