Senior AI Eval Engineer — LLM Judge & Data Pipelines

DroneDeploy, Inc.

Austin (TX)

Hybrid

USD 141,000 - 194,000

Full time

14 days+

A complete application in a minute

Tailored resume and cover letter, ready to send.

Get past ATS filters

Benefits offered by this job

Equity compensation
Bonus incentive

Job summary

Procore is seeking a Senior AI Engineer for its Evals & Goldsets team in Austin, TX. You will help measure and improve AI agent quality, building production traces and goldsets for evals, and turning traces into improvements.

This hybrid role reports to the Engineering Manager for Evals & Goldsets. You’ll collaborate with data and ML engineers to enhance speed, cost, reliability, and trust in our evals, balancing hands-on building with architectural review.

Qualifications

  • 5+ years of experience in software, AI, or ML engineering.
  • Strong proficiency in Python; experience with TypeScript is a plus.
  • Experience evaluating LLM systems, such as LLM-as-judge, rubric design, goldset curation, or measuring judge agreement.
  • Experience with LLM observability or evaluation tools such as Langfuse, Braintrust, or similar.
  • Experience building data pipelines, ETL processes, or streaming data systems.
  • Experience with analytics platforms and data warehouses such as Databricks or Snowflake.

Responsibilities

  • Improve our online evals: LLM judge accuracy, calibration, and the infrastructure that runs it in production.
  • Build and maintain production trace infrastructure with Langfuse and Databricks.
  • Build and maintain data pipelines for the goldsets behind our evals, including annotation and quality checks.
  • Build agentic loops that mine production traces for failures and turn them into new eval cases and agent improvements.
  • Improve the runtime, cost, and reliability of our online and offline eval pipelines.
  • Contribute code and architecture, review designs, and mentor other engineers on the team.

Skills

Python
TypeScript
LLM evaluation
LLM observability
Data pipelines

Education

Degree in computer science or related technical field

Tools

Langfuse
Databricks
Snowflake
Braintrust

Job description

Procore is seeking a Senior AI Engineer for its Evals & Goldsets team in Austin, TX. You will help measure and improve AI agent quality, building production traces and goldsets for evals, and turning traces into improvements.

This hybrid role reports to the Engineering Manager for Evals & Goldsets. You’ll collaborate with data and ML engineers to enhance speed, cost, reliability, and trust in our evals, balancing hands-on building with architectural review.

Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Senior AI Engineer - Evals & Goldsets Lead
Senior AI Engineer - Evals & Goldsets Lead

Procore Technologies, Inc. • Austin (TX)

Hybrid
USD 141,000 - 194,000
Equity compensation
Bonus incentive compensation
Senior AI Engineer - AI Eval & Goldsets (Hybrid, Austin)
Senior AI Engineer - AI Eval & Goldsets (Hybrid, Austin)

Procore Technologies • Austin (TX)

Hybrid
USD 141,000 - 194,000
Equity compensation
Base salary competitor
Senior AI Engineer
Senior AI Engineer

DroneDeploy • Austin (TX)

Hybrid
USD 141,000 - 194,000
Equity compensation
Bonus incentive
Senior AI Engineer
Senior AI Engineer

Procore Technologies • Austin (TX)

Hybrid
USD 141,000 - 194,000
Equity compensation
Base salary competitor
Senior AI Engineer
Senior AI Engineer

Procore Technologies, Inc. • Austin (TX)

Hybrid
USD 141,000 - 194,000
Equity compensation
Bonus incentive compensation
Senior AI Engineer – LLMs, RAG & Agent Workflows
Senior AI Engineer – LLMs, RAG & Agent Workflows

Procore Technologies, Inc. • Austin (TX)

On-site
USD 141,000 - 194,000
Senior AI Experience Engineer (Austin)
Senior AI Experience Engineer (Austin)

Procore Technologies • Austin (TX)

On-site
USD 141,000 - 194,000
Senior AI Engineer — LLM Evaluation & Production Systems
Senior AI Engineer — LLM Evaluation & Production Systems

LawPro.ai • Virginia (MN)

On-site
USD 140,000 - 200,000
AI Engineer (Data and Evaluations)
AI Engineer (Data and Evaluations)

Glean • San Francisco (CA)

On-site
USD 150,000 - 210,000
Senior AI Evaluation & Benchmark Engineer (Hybrid)
Senior AI Evaluation & Benchmark Engineer (Hybrid)

Glean • San Francisco (CA)

Hybrid
USD 200,000 - 280,000
Medical, Vision, Dental
Generous time-off
401k plan
+4