Production LLM/GenAI Engineer - RAG, Agents, & Optimization

Evlo AI; New York (NY); On-site; USD 150,000 - 230,000; Full time

Job summary

Evlo AI is seeking an experienced software engineer to design and ship production LLM systems that handle real user traffic. You will own retrieval-augmented generation pipelines, agentic workflows, fine-tuning jobs, and evaluation infrastructure, balancing latency, cost, and accuracy.

You will collaborate with applied scientists, backend engineers, and product leadership to turn frontier model capabilities into revenue‑generating features.

Qualifications

• 3–7 years of software engineering experience, including 2+ years building LLM/GenAI systems in production.
• Deep hands-on experience with at least one LLM orchestration framework (LangChain, LlamaIndex, Haystack) and the judgment to know when not to use one
• Production experience with vector databases, embedding models, and semantic search at scale
• Strong Python engineering skills: async programming, API design, testing, and cloud infrastructure (AWS, GCP, or Azure)
• Experience fine-tuning or evaluating LLMs (LoRA/QLoRA, instruction tuning, RLHF concepts) and a rigorous approach to measuring model quality
• MS or PhD in Computer Science, ML, or related field, or equivalent industry experience
• Bonus: experience serving models with vLLM or TensorRT‑LLM, GPU cost optimization, contributions to open‑source LLM tooling, or building eval infrastructure for agentic systems

Responsibilities

• Design, build, and maintain production RAG pipelines using LangChain, LlamaIndex, or custom retrieval architectures with hybrid search and re‑ranking
• Optimize vector database performance (Pinecone, Weaviate, pgvector) for low‑latency semantic retrieval across large‑scale document corpora
• Build agentic systems with tool use, multi‑step reasoning, and guardrails — including orchestration, state management, and failure recovery
• Develop and run systematic evaluation frameworks: golden datasets, LLM‑as‑judge pipelines, regression suites, and online A/B tests tied to product metrics
• Fine‑tune open‑weight and frontier models using LoRA/QLoRA and supervised instruction tuning on domain‑specific datasets
• Instrument LLM applications for observability — tracing, token cost tracking, drift detection — and drive improvements in quality, latency, and cost per request
• Contribute to architecture reviews and establish engineering standards for shipping AI features safely to production

Skills

Python

LLM/GenAI systems

LangChain/LlamaIndex

Vector databases

Async programming

Cloud infrastructure

Education

MS/PhD in CS/ML

Tools

LangChain

LlamaIndex

Haystack

Pinecone

Weaviate

pgvector

TensorRT-LLM

vLLM

Job description

Evlo AI is seeking an experienced software engineer to design and ship production LLM systems that handle real user traffic. You will own retrieval-augmented generation pipelines, agentic workflows, fine-tuning jobs, and evaluation infrastructure, balancing latency, cost, and accuracy.

You will collaborate with applied scientists, backend engineers, and product leadership to turn frontier model capabilities into revenue‑generating features.