ML Engineer: End-to-End AI Pipelines & Evaluation

ESB Technologies; United States; On-site; USD 150,000 - 210,000; Full time

Job summary

ESB Technologies seeks an experienced ML/LLM systems engineer to build production-grade pipelines that turn client interview data into structured process maps and SOPs. You will design retrieval and evaluation mechanisms across Claude, ChatGPT, Gemini and Slack, and implement autonomy scoring to optimize workflow automation.

You will work with founding engineers to ship end-to-end in production, balancing cost, latency and quality while ensuring privacy and tenant isolation across client

Qualifications

• 5+ years building ML or LLM systems that run in production with real users.
• Strong Python and sound engineering habits: testing, versioning, observability.
• Hands-on LLM experience: context design, structured output, tool use, retrieval, agents.
• A record of building the evaluation before trusting the model change.
• Experience with unstructured input: transcripts, audio, documents, screenshots.
• Working statistics: design a measurement, size a sample, and interpret results.
• Comfort with ambiguity and with direct contact with customers and founders.

Responsibilities

• Own the pipeline from interview transcripts, screen recordings and uploaded files to structured process maps and draft SOPs.
• Build the evaluation harness: golden sets, regression tests, and scoring for extraction accuracy and SOP quality.
• Design retrieval over client workflows so the assistant answers consistently across Claude, ChatGPT, Gemini and Slack.
• Build the scoring behind the autonomy ladder and determine when to auto-publish.
• Detect deviations from SOPs, classify, and route retries and human review.
• Improve time and cost estimates per workflow with clear labeling as estimates.
• Choose models per task based on cost, latency and quality; maintain portability across providers.
• Ensure tenant isolation and privacy in every pipeline.

Skills

Python

LLM systems in production

Testing & observability

Customer interaction

Ambiguity tolerance

Tools

Claude

ChatGPT

Gemini

Slack

Job description

ESB Technologies seeks an experienced ML/LLM systems engineer to build production-grade pipelines that turn client interview data into structured process maps and SOPs. You will design retrieval and evaluation mechanisms across Claude, ChatGPT, Gemini and Slack, and implement autonomy scoring to optimize workflow automation.

You will work with founding engineers to ship end-to-end in production, balancing cost, latency and quality while ensuring privacy and tenant isolation across client