LLM / GenAI Engineer

Evlo AI; New York (NY); On-site; USD 150,000 - 230,000; Full time

Job summary

Evlo AI is seeking an experienced software engineer to design and ship production LLM systems that handle real user traffic. You will own retrieval-augmented generation pipelines, agentic workflows, fine-tuning jobs, and evaluation infrastructure, balancing latency, cost, and accuracy.

You will collaborate with applied scientists, backend engineers, and product leadership to turn frontier model capabilities into revenue‑generating features.

Qualifications

• 3–7 years of software engineering experience, including 2+ years building LLM/GenAI systems in production.
• Deep hands-on experience with at least one LLM orchestration framework (LangChain, LlamaIndex, Haystack) and the judgment to know when not to use one
• Production experience with vector databases, embedding models, and semantic search at scale
• Strong Python engineering skills: async programming, API design, testing, and cloud infrastructure (AWS, GCP, or Azure)
• Experience fine-tuning or evaluating LLMs (LoRA/QLoRA, instruction tuning, RLHF concepts) and a rigorous approach to measuring model quality
• MS or PhD in Computer Science, ML, or related field, or equivalent industry experience
• Bonus: experience serving models with vLLM or TensorRT‑LLM, GPU cost optimization, contributions to open‑source LLM tooling, or building eval infrastructure for agentic systems

Responsibilities

• Design, build, and maintain production RAG pipelines using LangChain, LlamaIndex, or custom retrieval architectures with hybrid search and re‑ranking
• Optimize vector database performance (Pinecone, Weaviate, pgvector) for low‑latency semantic retrieval across large‑scale document corpora
• Build agentic systems with tool use, multi‑step reasoning, and guardrails — including orchestration, state management, and failure recovery
• Develop and run systematic evaluation frameworks: golden datasets, LLM‑as‑judge pipelines, regression suites, and online A/B tests tied to product metrics
• Fine‑tune open‑weight and frontier models using LoRA/QLoRA and supervised instruction tuning on domain‑specific datasets
• Instrument LLM applications for observability — tracing, token cost tracking, drift detection — and drive improvements in quality, latency, and cost per request
• Contribute to architecture reviews and establish engineering standards for shipping AI features safely to production

Skills

Python

LLM/GenAI systems

LangChain/LlamaIndex

Vector databases

Async programming

Cloud infrastructure

Education

MS/PhD in CS/ML

Tools

LangChain

LlamaIndex

Haystack

Pinecone

Weaviate

pgvector

TensorRT-LLM

vLLM

Job description

About The Role

The role builds and ships production LLM systems that handle real user traffic: retrieval-augmented generation pipelines, agentic workflows, fine-tuning jobs, and evaluation infrastructure. This is hands‑on engineering work, not prompt tinkering — latency budgets, cost controls, and hallucination rates are the metrics that matter.

The team sits at the center of the product, working directly with applied scientists, backend engineers, and product leadership to turn frontier model capabilities into reliable, revenue‑generating features. Decisions made here shape how the entire organization uses AI.

Key Responsibilities

• Design, build, and maintain production RAG pipelines using LangChain, LlamaIndex, or custom retrieval architectures with hybrid search and re‑ranking
• Optimize vector database performance (Pinecone, Weaviate, pgvector) for low‑latency semantic retrieval across large‑scale document corpora
• Build agentic systems with tool use, multi‑step reasoning, and guardrails — including orchestration, state management, and failure recovery
• Develop and run systematic evaluation frameworks: golden datasets, LLM‑as‑judge pipelines, regression suites, and online A/B tests tied to product metrics
• Fine‑tune open‑weight and frontier models using LoRA/QLoRA and supervised instruction tuning on domain‑specific datasets
• Instrument LLM applications for observability — tracing, token cost tracking, drift detection — and drive improvements in quality, latency, and cost per request
• Contribute to architecture reviews and establish engineering standards for shipping AI features safely to production

What We Are Looking For

• 3–7 years of software engineering experience, including 2+ years building LLM/GenAI systems in production
• Deep hands‑on experience with at least one LLM orchestration framework (LangChain, LlamaIndex, Haystack) and the judgment to know when not to use one
• Production experience with vector databases, embedding models, and semantic search at scale
• Strong Python engineering skills: async programming, API design, testing, and cloud infrastructure (AWS, GCP, or Azure)
• Experience fine‑tuning or evaluating LLMs (LoRA/QLoRA, instruction tuning, RLHF concepts) and a rigorous approach to measuring model quality
• MS or PhD in Computer Science, ML, or related field, or equivalent industry experience
• Bonus: experience serving models with vLLM or TensorRT‑LLM, GPU cost optimization, contributions to open‑source LLM tooling, or building eval infrastructure for agentic systems