Senior AI Engineer, Eval & Goldsets (Hybrid - Austin)

Procore

Jackson (TN)

Hybrid

USD 141,000 - 194,000

Full time

14 days+

A complete application in a minute

Tailored resume and cover letter, ready to send.

Get past ATS filters

Benefits offered by this job

Equity compensation
Bonus incentive

Job summary

Procore is seeking a Senior AI Engineer to join the Evals & Goldsets team. You will partner with data and ML engineers to improve evals, reduce costs, and make the system trustworthy as we measure AI agents for the construction industry.

This hybrid role reports to the Engineering Manager for Evals & Goldsets and is based in Austin, TX. You will own production traces, build data pipelines, and mentor peers while shaping how AI in construction is tested and improved.

Qualifications

  • 5+ years in software, AI, or ML engineering.
  • Proficiency in Python; TypeScript is a plus.
  • Experience evaluating LLM systems and rubrics.
  • Experience with Langfuse, Databricks, or similar tools.
  • Ability to design data pipelines and data warehouses.

Responsibilities

  • Improve online evals: judge accuracy and production infrastructure.
  • Build and maintain production trace infra with Langfuse and Databricks.
  • Create data pipelines for goldsets and evals with quality checks.
  • Develop agentic loops to mine traces into new eval cases.
  • Mentor engineers and influence teams with eval results.

Skills

Python
TypeScript
LLM evaluation
ETL pipelines
Data-driven decision making

Education

Bachelor's degree in CS or related field

Tools

Langfuse
Databricks
Snowflake

Job description

Procore is seeking a Senior AI Engineer to join the Evals & Goldsets team. You will partner with data and ML engineers to improve evals, reduce costs, and make the system trustworthy as we measure AI agents for the construction industry.

This hybrid role reports to the Engineering Manager for Evals & Goldsets and is based in Austin, TX. You will own production traces, build data pipelines, and mentor peers while shaping how AI in construction is tested and improved.

Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Senior AI Engineer - Evals & Goldsets Lead
Senior AI Engineer - Evals & Goldsets Lead

Procore Technologies, Inc. • Austin (TX)

Hybrid
USD 141,000 - 194,000
Equity compensation
Bonus incentive compensation
Senior AI Engineer - AI Eval & Goldsets (Hybrid, Austin)
Senior AI Engineer - AI Eval & Goldsets (Hybrid, Austin)

Procore Technologies, Inc. • Austin (TX)

Hybrid
USD 141,000 - 194,000
Equity compensation
Base salary competitor
Senior AI Eval Engineer — LLM Judge & Data Pipelines
Senior AI Eval Engineer — LLM Judge & Data Pipelines

DroneDeploy, Inc. • Austin (TX)

Hybrid
USD 141,000 - 194,000
Equity compensation
Bonus incentive
Staff AI Engineer - Lead GenAI & Scalable Systems (Hybrid)
Staff AI Engineer - Lead GenAI & Scalable Systems (Hybrid)

Procore • Jackson (TN)

Hybrid
USD 169,000 - 232,000
Staff AI Engineer: GenAI & Scalable Production Systems
Staff AI Engineer: GenAI & Scalable Production Systems

Procore Technologies, Inc. • United States

Hybrid
USD 169,000 - 232,000
Equity Compensation
Bonus Incentive Compensation
Senior AI Experience Engineer (Austin)
Senior AI Experience Engineer (Austin)

Procore Technologies, Inc. • Austin (TX)

On-site
USD 141,000 - 194,000
Staff AI Engineer - Generative AI Leader (Hybrid TX)
Staff AI Engineer - Generative AI Leader (Hybrid TX)

DroneDeploy, Inc. • Austin (TX)

Hybrid
USD 169,000 - 232,000
Senior AI Engineer
Senior AI Engineer

Procore Technologies, Inc. • Austin (TX)

Hybrid
USD 141,000 - 194,000
Equity compensation
Base salary competitor
Senior AI Engineer
Senior AI Engineer

DroneDeploy • Austin (TX)

Hybrid
USD 141,000 - 194,000
Equity compensation
Bonus incentive
Senior AI Software Engineer - Construction Platform
Senior AI Software Engineer - Construction Platform

DroneDeploy, Inc. • Austin (TX)

On-site
USD 141,000 - 194,000
Equity compensation
Bonus incentive compensation
Competitive compensation