AI Platform Engineer for LLM Ops

LTS; United States; Remote; USD 150,000 - 230,000; Full time

Job summary

LTS seeks an AI Platform and Harness Engineer to design, build, and maintain the enterprise AI platform, AI harnesses, and evaluation frameworks powering LLMs, agents, and RAG deployments at scale.

The ideal candidate has 5+ years of software/ platform engineering experience, strong Python skills, and proficiency with cloud-native backend systems, observability, and CI/CD. You will collaborate across AI architects, engineers, data scientists, and product teams.

Qualifications

• Bachelor's degree in Computer Science, Software Engineering, Artificial Intelligence, Data Science, or a related technical field.
• 5+ years of experience in software engineering, platform engineering, backend engineering, DevOps, cloud engineering, or infrastructure engineering.
• 2+ years building or supporting Generative AI, Large Language Model (LLM), or machine learning applications.
• Strong programming experience in Python.
• Experience developing APIs, backend services, and distributed systems.
• Experience with cloud platforms including AWS, Azure, or Google Cloud Platform.
• Experience deploying applications using Docker and Kubernetes.
• Experience working with Git, CI/CD pipelines, Infrastructure as Code (IaC), and infrastructure automation.
• Strong understanding of Large Language Models (LLMs), Retrieval-Augmented Generation (RAG), Prompt engineering, Embeddings, Vector databases, AI agents and agentic workflows
• Familiarity with AI evaluation techniques, automated testing, benchmarking, regression testing, and model validation.
• Experience building scalable, production-grade software platforms.
• Strong problem-solving, debugging, and performance optimization skills.

Responsibilities

• Design, build, and maintain enterprise AI platform capabilities supporting LLMs, AI agents, RAG, and GenAI apps.
• Develop reusable AI harnesses to automate testing, prompt evaluation, model benchmarking, regression testing, and QA.
• Build AI evaluation frameworks to measure model accuracy, retrieval quality, latency, cost, and performance.
• Implement observability and monitoring solutions for AI applications, including telemetry, tracing, logging, dashboards, and metrics.
• Build and maintain LLMOps pipelines for model deployment, versioning, evaluation, experimentation, rollback, and improvement.
• Design automated workflows for prompt testing, retrieval evaluation, AI system validation, and performance benchmarking.
• Develop internal tools for prompt management, model experimentation, AI performance optimization, and developer productivity.
• Build scalable backend services and APIs supporting AI platforms and enterprise AI integrations.
• Collaborate with AI architects and engineering teams to integrate LLMs, RAG pipelines, vector databases, and agentic AI solutions.
• Support deployment of AI services across AWS, Azure, or Google Cloud using containerized and cloud-native architectures.
• Implement CI/CD pipelines and infrastructure automation for enterprise AI development and deployment.
• Apply security, governance, and Responsible AI controls throughout the lifecycle.
• Evaluate emerging AI frameworks and tools to improve engineering productivity.
• Troubleshoot production AI issues and continuously improve platform reliability, scalability, and user experience.
• Document engineering standards, AI platform architecture, evaluation methodologies, and operational best practices.

Skills

Python

Problem solving

Debugging

Performance optimization

Education

Bachelor's degree in Computer Science/Software Engineering/Artificial Intelligence/Data Science

Tools

Docker

Kubernetes

Git

CI/CD pipelines

Infrastructure as Code (IaC)

LangChain

LangGraph

LlamaIndex

Semantic Kernel

AutoGen

Prometheus

Grafana

Job description

LTS seeks an AI Platform and Harness Engineer to design, build, and maintain the enterprise AI platform, AI harnesses, and evaluation frameworks powering LLMs, agents, and RAG deployments at scale.

The ideal candidate has 5+ years of software/ platform engineering experience, strong Python skills, and proficiency with cloud-native backend systems, observability, and CI/CD. You will collaborate across AI architects, engineers, data scientists, and product teams.