Senior GenAI & LLM Evaluation Engineer - Remote US

Acuity Analytics; United States; On-site; USD 140,000 - 190,000; Full time

Job summary

Acuity Analytics is seeking a Senior AI & Data Engineer to lead GenAI and LLM evaluation work, partnering with data scientists and engineers. You will design scalable data pipelines, develop evaluation frameworks, and deploy production AI solutions, shaping high-impact data products for diverse clients.

The role emphasizes hands-on implementation, collaboration in Agile teams, and delivering robust AI-enabled data platforms within US time zones. Join a growth-focused, client-facing environment.

Qualifications

• 5+ years' experience in Data Engineering, Machine Learning, AI Engineering, or Data Science.
• Strong Python and SQL skills.
• Hands-on experience with Generative AI and Large Language Models (LLMs).
• Experience evaluating LLMs using golden datasets, model benchmarking, or LLM-as-a-Judge techniques.
• Experience building and deploying production-ready ML solutions.
• Strong understanding of data pipelines, validation, and quality frameworks.
• Experience integrating APIs and cloud-based AI services.
• Strong problem-solving and analytical skills.
• Excellent communication and stakeholder management.
• Experience working in Agile, cross-functional teams.
• Consulting or customer-facing experience is a plus.

Responsibilities

• Build and maintain scalable data pipelines and workflows.
• Develop evaluation frameworks for GenAI and LLM applications.
• Create and maintain golden datasets for model testing.
• Evaluate and benchmark LLM performance.
• Build ML-assisted tools to improve delivery and automation.
• Deploy production-ready AI solutions.
• Work closely with delivery teams to solve technical challenges.
• Continuously improve data quality, automation, and operational efficiency.

Skills

Python

SQL

Generative AI

LLMs

LLM evaluation

Production ML

Data pipelines

APIs / Cloud

Analytical skills

Communication

Agile teamwork

Consulting / Customer facing

Job description

Acuity Analytics is seeking a Senior AI & Data Engineer to lead GenAI and LLM evaluation work, partnering with data scientists and engineers. You will design scalable data pipelines, develop evaluation frameworks, and deploy production AI solutions, shaping high-impact data products for diverse clients.

The role emphasizes hands-on implementation, collaboration in Agile teams, and delivering robust AI-enabled data platforms within US time zones. Join a growth-focused, client-facing environment.