AI Platform & LLM Ops Engineer

Doist; United States; Remote; USD 140,000 - 230,000; Full time

Benefits offered by this job

Comprehensive benefits

Career growth opportunities

Cutting-edge tools and technologies

Job summary

LTS is seeking an AI Platform and Harness Engineer to design, build, and maintain an enterprise AI platform powering LLMs, AI agents, and Generative AI apps. You will create reusable AI harnesses, evaluation frameworks, and observability tooling to securely develop, test, and deploy AI at scale.

You will collaborate with AI architects, data scientists, and engineers to ensure scalable, cost-effective solutions with strong security and governance practices.

Qualifications

• Bachelor's degree in a related technical field or equiv.
• 5+ years of software/platform engineering experience.
• 2+ years building Generative AI or LLM-based apps.
• Strong Python programming proficiency.
• Experience with cloud platforms (AWS/Azure/Google).
• Docker and Kubernetes deployment experience.
• Experience with CI/CD and Infrastructure as Code.

Responsibilities

• Design, build, and maintain enterprise AI platform capabilities for LLMs, AI agents, RAG, and Generative AI apps.
• Develop reusable AI harnesses to automate testing, prompt evaluation, and model benchmarking.
• Build evaluation frameworks measuring accuracy, latency, cost, and performance.
• Implement observability and monitoring for AI apps with telemetry and dashboards.
• Maintain LLMOps pipelines for deployment, versioning, evaluation, and experiments.
• Automate prompts testing, retrieval evaluation, and system validation workflows.
• Create internal tools for prompt management, model experimentation, and performance optimization.
• Develop scalable backend services and APIs for AI platforms and enterprise integrations.
• Collaborate with architects/engineers to integrate LLMs, RAG pipelines, and vector databases.
• Support deployment across AWS/Azure/Google Cloud using containerized architectures.
• Implement CI/CD and infrastructure automation; ensure security and governance.
• Evaluate emerging AI frameworks and tooling to improve productivity.
• Troubleshoot production AI issues and improve platform reliability.

Skills

Backend engineering

Python

Cloud engineering

DevOps

CI/CD

IaC

Docker

Kubernetes

LLMOps

Observability

Education

Bachelor's degree

Tools

Git

LangChain (nice-to-have)

OpenTelemetry (nice-to-have)

Job description

LTS is seeking an AI Platform and Harness Engineer to design, build, and maintain an enterprise AI platform powering LLMs, AI agents, and Generative AI apps. You will create reusable AI harnesses, evaluation frameworks, and observability tooling to securely develop, test, and deploy AI at scale.

You will collaborate with AI architects, data scientists, and engineers to ensure scalable, cost-effective solutions with strong security and governance practices.