Senior AI Engineer - Evals & Goldsets Lead

Procore Technologies, Inc.

Austin (TX)

Hybrid

USD 141,000 - 194,000

Full time

14 days+

A complete application in a minute

Tailored resume and cover letter, ready to send.

Get past ATS filters

Benefits offered by this job

Equity compensation
Bonus incentive compensation

Job summary

Procore Technologies, Inc. in Austin, TX is seeking a Senior AI Engineer to join the Evals & Goldsets team, responsible for measuring and improving the quality of our AI agents in the construction domain.

You'll partner with data and ML engineers to accelerate evals, reduce costs, and increase reliability by turning production traces into actionable feedback loops for improved agent behavior. This hybrid role reports to the Engineering Manager.

Qualifications

  • 5+ years of experience in software, AI, or ML engineering.
  • Strong proficiency in Python; experience with TypeScript is a plus.
  • Experience evaluating LLM systems, such as LLM-as-judge, rubric design, goldset curation, or measuring judge agreement.
  • Experience with LLM observability or evaluation tools such as Langfuse, Braintrust, or similar.
  • Experience building data pipelines, ETL processes, or streaming data systems.
  • Experience with analytics platforms and data warehouses such as Databricks or Snowflake.
  • Ability to communicate eval results clearly and influence the teams that build on them.
  • Degree in computer science or a related technical field, or equivalent practical experience.

Responsibilities

  • Improve our online evals: LLM judge accuracy, calibration, and the infrastructure that runs it in production.
  • Build and maintain production trace infrastructure with Langfuse and Databricks.
  • Build and maintain data pipelines for the goldsets behind our evals, including annotation and quality checks.
  • Build agentic loops that mine production traces for failures and turn them into new eval cases and agent improvements.
  • Improve the runtime, cost, and reliability of our online and offline eval pipelines.
  • Contribute code and architecture, review designs, and mentor other engineers on the team.

Skills

Python
TypeScript
LLM evaluation
Langfuse
Databricks
ETL pipelines
Data warehouses
Communication

Education

Degree in computer science or related field

Tools

Langfuse
Databricks
Snowflake

Job description

Procore Technologies, Inc. in Austin, TX is seeking a Senior AI Engineer to join the Evals & Goldsets team, responsible for measuring and improving the quality of our AI agents in the construction domain.

You'll partner with data and ML engineers to accelerate evals, reduce costs, and increase reliability by turning production traces into actionable feedback loops for improved agent behavior. This hybrid role reports to the Engineering Manager.

Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Senior AI Engineer - AI Eval & Goldsets (Hybrid, Austin)
Senior AI Engineer - AI Eval & Goldsets (Hybrid, Austin)

Procore Technologies • Austin (TX)

Hybrid
USD 141,000 - 194,000
Equity compensation
Base salary competitor
Senior AI Eval Engineer — LLM Judge & Data Pipelines
Senior AI Eval Engineer — LLM Judge & Data Pipelines

DroneDeploy, Inc. • Austin (TX)

Hybrid
USD 141,000 - 194,000
Equity compensation
Bonus incentive
Senior AI Experience Engineer (Austin)
Senior AI Experience Engineer (Austin)

Procore Technologies • Austin (TX)

On-site
USD 141,000 - 194,000
Senior AI Engineer – LLMs, RAG & Agent Workflows
Senior AI Engineer – LLMs, RAG & Agent Workflows

Procore Technologies, Inc. • Austin (TX)

On-site
USD 141,000 - 194,000
Staff AI Engineer - Generative AI Leader (Hybrid TX)
Staff AI Engineer - Generative AI Leader (Hybrid TX)

DroneDeploy, Inc. • Austin (TX)

Hybrid
USD 169,000 - 232,000
Staff AI Engineer: Scalable, Secure Production Systems
Staff AI Engineer: Scalable, Secure Production Systems

Procore Technologies, Inc. • Austin (TX)

Hybrid
USD 169,000 - 232,000
Senior AI Engineer
Senior AI Engineer

Procore Technologies • Austin (TX)

Hybrid
USD 141,000 - 194,000
Equity compensation
Base salary competitor
Senior AI Software Engineer - Construction Platform
Senior AI Software Engineer - Construction Platform

DroneDeploy, Inc. • Austin (TX)

On-site
USD 141,000 - 194,000
Equity compensation
Bonus incentive compensation
Competitive compensation
Senior AI Engineer
Senior AI Engineer

Procore Technologies, Inc. • Austin (TX)

Hybrid
USD 141,000 - 194,000
Equity compensation
Bonus incentive compensation
Senior AI Engineer
Senior AI Engineer

DroneDeploy • Austin (TX)

Hybrid
USD 141,000 - 194,000
Equity compensation
Bonus incentive