Senior AI Engineer - AI Eval & Goldsets (Hybrid, Austin)

Procore Technologies

Austin (TX)

Hybrid

USD 141,000 - 194,000

Full time

14 days+

Get a reply from this employer

Your resume and cover letter, tailored to exactly what they’re hiring for.

Get past ATS filters

Benefits offered by this job

Equity compensation
Base salary competitor

Job summary

Procore Technologies in Austin, TX is seeking a Senior AI Engineer to join the Evals & Goldsets team, building and running systems that measure AI agents and their performance in production.

You will partner with data and ML engineers to make evals faster, cheaper, more trustworthy, and harder to saturate, turning production traces into feedback loops that improve agent behavior. This is a hybrid role reporting to the Engineering Manager in Austin.

Qualifications

  • 5+ years of experience in software, AI, or ML engineering.
  • Strong proficiency in Python; TypeScript is a plus.
  • Experience evaluating LLM systems and measuring judge agreement.
  • Experience with LLM observability or evaluation tools such as Langfuse, Braintrust, or similar.
  • Experience building data pipelines, ETL processes, or streaming data systems.
  • Experience with analytics platforms and data warehouses such as Databricks or Snowflake.
  • Ability to communicate eval results clearly and influence teams.
  • Degree in computer science or related technical field, or equivalent practical experience.

Responsibilities

  • Improve our online evals: LLM judge accuracy, calibration, and the infrastructure that runs it in production.
  • Build and maintain production trace infrastructure with Langfuse and Databricks.
  • Build and maintain data pipelines for the goldsets behind our evals, including annotation and quality checks.
  • Build agentic loops that mine production traces for failures and turn them into new eval cases and agent improvements.
  • Improve the runtime, cost, and reliability of our online and offline eval pipelines.
  • Contribute code and architecture, review designs, and mentor other engineers on the team.

Skills

5+ yrs experience
Python
LLM evaluation
LLM observability
Data pipelines
Databricks
Snowflake
Communication

Education

CS degree or equivalent

Tools

Langfuse
Databricks (knowledge)

Job description

Procore Technologies in Austin, TX is seeking a Senior AI Engineer to join the Evals & Goldsets team, building and running systems that measure AI agents and their performance in production.

You will partner with data and ML engineers to make evals faster, cheaper, more trustworthy, and harder to saturate, turning production traces into feedback loops that improve agent behavior. This is a hybrid role reporting to the Engineering Manager in Austin.

Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Senior AI Engineer - Evals & Goldsets Lead
Senior AI Engineer - Evals & Goldsets Lead

Procore Technologies, Inc. • Austin (TX)

Hybrid
USD 141,000 - 194,000
Equity compensation
Bonus incentive compensation
Senior AI Eval Engineer — LLM Judge & Data Pipelines
Senior AI Eval Engineer — LLM Judge & Data Pipelines

DroneDeploy, Inc. • Austin (TX)

Hybrid
USD 141,000 - 194,000
Equity compensation
Bonus incentive
Senior AI Experience Engineer (Austin)
Senior AI Experience Engineer (Austin)

Procore Technologies • Austin (TX)

On-site
USD 141,000 - 194,000
Staff AI Engineer - Generative AI Leader (Hybrid TX)
Staff AI Engineer - Generative AI Leader (Hybrid TX)

DroneDeploy, Inc. • Austin (TX)

Hybrid
USD 169,000 - 232,000
Staff AI Engineer: Scalable, Secure Production Systems
Staff AI Engineer: Scalable, Secure Production Systems

Procore Technologies, Inc. • Austin (TX)

Hybrid
USD 169,000 - 232,000
Senior AI Evaluation Engineer (Hybrid: Bay Area)
Senior AI Evaluation Engineer (Hybrid: Bay Area)

Glean • Mountain View (CA)

Hybrid
USD 200,000 - 280,000
Medical, Vision, Dental
401k plan
Home office stipend
+3
Senior AI Engineer
Senior AI Engineer

Procore Technologies • Austin (TX)

Hybrid
USD 141,000 - 194,000
Equity compensation
Base salary competitor
AI Evaluation Engineer: RL Environments & Agents
AI Evaluation Engineer: RL Environments & Agents

MaxIT Consulting - Max Corporate Group • San Francisco (CA)

On-site
USD 140,000 - 210,000
Senior AI Engineer
Senior AI Engineer

DroneDeploy • Austin (TX)

Hybrid
USD 141,000 - 194,000
Equity compensation
Bonus incentive
Senior AI Engineer
Senior AI Engineer

Procore Technologies, Inc. • Austin (TX)

Hybrid
USD 141,000 - 194,000
Equity compensation
Bonus incentive compensation