Senior AI Engineer

DroneDeploy

Austin (TX)

Hybrid

USD 141,000 - 194,000

Full time

14 days+

A complete application in a minute

Tailored resume and cover letter, ready to send.

Get past ATS filters

Benefits offered by this job

Equity compensation
Bonus incentive

Job summary

Procore is seeking a Senior AI Engineer for its Evals & Goldsets team in Austin, TX. You will help measure and improve AI agent quality, building production traces and goldsets for evals, and turning traces into improvements.

This hybrid role reports to the Engineering Manager for Evals & Goldsets. You’ll collaborate with data and ML engineers to enhance speed, cost, reliability, and trust in our evals, balancing hands-on building with architectural review.

Qualifications

  • 5+ years of experience in software, AI, or ML engineering.
  • Strong proficiency in Python; experience with TypeScript is a plus.
  • Experience evaluating LLM systems, such as LLM-as-judge, rubric design, goldset curation, or measuring judge agreement.
  • Experience with LLM observability or evaluation tools such as Langfuse, Braintrust, or similar.
  • Experience building data pipelines, ETL processes, or streaming data systems.
  • Experience with analytics platforms and data warehouses such as Databricks or Snowflake.

Responsibilities

  • Improve our online evals: LLM judge accuracy, calibration, and the infrastructure that runs it in production.
  • Build and maintain production trace infrastructure with Langfuse and Databricks.
  • Build and maintain data pipelines for the goldsets behind our evals, including annotation and quality checks.
  • Build agentic loops that mine production traces for failures and turn them into new eval cases and agent improvements.
  • Improve the runtime, cost, and reliability of our online and offline eval pipelines.
  • Contribute code and architecture, review designs, and mentor other engineers on the team.

Skills

Python
TypeScript
LLM evaluation
LLM observability
Data pipelines

Education

Degree in computer science or related technical field

Tools

Langfuse
Databricks
Snowflake
Braintrust

Job description

We're looking for a Senior AI Engineer to join Procore's Evals & Goldsets team, which measures and improves the quality of the AI agents we ship to the construction industry. You'll build and run the systems that tell our AI teams how well their agents are doing: production tracing in Langfuse and Databricks, our online LLM judge, and the goldsets and benchmarks behind our offline evals.

As a Senior AI Engineer, you will partner with data and ML engineers to make our evals faster, cheaper, more trustworthy, and harder to saturate, and you'll turn production traces into feedback loops that improve agent behavior. We're a small, fast-moving team that pairs hands-on building with architecture and review, and we value engineers who are rigorous about measurement and comfortable with ambiguity. If you want to shape how AI in construction gets tested and improved, come join us.

This hybrid position reports to the Engineering Manager for Evals & Goldsets and will be based in our Austin, TX office.

What you’ll do:

  • Improve our online evals: LLM judge accuracy, calibration, and the infrastructure that runs it in production.
  • Build and maintain production trace infrastructure with Langfuse and Databricks.
  • Build and maintain data pipelines for the goldsets behind our evals, including annotation and quality checks.
  • Build agentic loops that mine production traces for failures and turn them into new eval cases and agent improvements.
  • Improve the runtime, cost, and reliability of our online and offline eval pipelines.
  • Contribute code and architecture, review designs, and mentor other engineers on the team.

What we’re looking for:

  • 5+ years of experience in software, AI, or ML engineering.
  • Strong proficiency in Python; experience with TypeScript is a plus.
  • Experience evaluating LLM systems, such as LLM-as-judge, rubric design, goldset curation, or measuring judge agreement.
  • Experience with LLM observability or evaluation tools such as Langfuse, Braintrust, or similar.
  • Experience building data pipelines, ETL processes, or streaming data systems.
  • Experience with analytics platforms and data warehouses such as Databricks or Snowflake.
  • Ability to communicate eval results clearly and influence the teams that build on them.
  • Degree in computer science or a related technical field, or equivalent practical experience.
Additional Information

Base Pay Range:

140,960.00 - 193,820.00 USD Annual

This role may also be eligible for Equity Compensation and/or Bonus Incentive Compensation. Procore is committed to offering competitive, fair, and commensurate compensation. Actual compensation will be based on a candidate’s job-related skills, experience, education or training, and location.

For Los Angeles County (unincorporated) Candidates:

Procore will consider for employment all qualified applicants, including those with arrest or conviction records, in accordance with the requirements of applicable federal, state, and local laws, including the City of Los Angeles’ Fair Chance Initiative for Hiring Ordinance, the Los Angeles County Fair Chance Ordinance for Employers, and the California Fair Chance Act.

A criminal history may have a direct, adverse, and negative relationship on the following job duties, potentially resulting in the withdrawal of the conditional offer of employment: 1. appropriately managing, accessing, and handling confidential information including proprietary and trade secret information, as well as accessing Procore's information technology systems and platforms; 2. interacting with and occasionally having unsupervised contact with internal/external customers, stakeholders, and/or colleagues; and 3. exercising sound judgment.

Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Senior AI Engineer
Senior AI Engineer

Procore Technologies, Inc. • Austin (TX)

Hybrid
USD 141,000 - 194,000
Equity compensation
Bonus incentive compensation
Senior AI Engineer
Senior AI Engineer

Procore Technologies • Austin (TX)

Hybrid
USD 141,000 - 194,000
Equity compensation
Base salary competitor
Senior Software Engineer - Procore AI Experience
Senior Software Engineer - Procore AI Experience

Procore • Austin (TX)

On-site
USD 141,000 - 194,000
Senior Software Engineer - Procore AI Experience
Senior Software Engineer - Procore AI Experience

DroneDeploy, Inc. • Austin (TX)

On-site
USD 141,000 - 194,000
Equity compensation
Bonus incentive compensation
Competitive compensation
Senior Software Engineer - Procore AI Experience
Senior Software Engineer - Procore AI Experience

Procore Technologies • Austin (TX)

On-site
USD 141,000 - 194,000
AI Application Architect
AI Application Architect

DroneDeploy, Inc. • Austin (TX)

On-site
USD 233,000 - 321,000
Equity compensation
Bonus incentives
Staff Software Engineer - AI
Staff Software Engineer - AI

DroneDeploy • Austin (TX)

Hybrid
USD 169,000 - 232,000
Senior Software Engineer, AI Platform
Senior Software Engineer, AI Platform

Procore • Austin (TX)

Hybrid
USD 141,000 - 194,000
Senior Principal Software Engineer, Ruby
Senior Principal Software Engineer, Ruby

Procore • Austin (TX)

Hybrid
USD 233,000 - 321,000
Equity Compensation
Bonus Incentive
Staff Software Engineer - AI
Staff Software Engineer - AI

Procore Technologies, Inc. • Austin (TX)

Hybrid
USD 169,000 - 232,000