Senior AI Engineer – LLM Evaluation & Data Pipelines

Procore Technologies, Inc.

Auckland

On-site

NZD 92,000 - 127,000

Full time

14 days+

A complete application in a minute

Tailored resume and cover letter, ready to send.

Get past ATS filters

Benefits offered by this job

Equity compensation
Bonus incentive compensation

Job summary

Procore Technologies, Inc. in Auckland is seeking a Senior Software Engineer to own the data, tooling, and evaluation infrastructure for AI-powered features.

You will build robust data pipelines, improve model quality, and drive prompts and models from candidate selection through production rollout and validation. You will collaborate with engineering, product, and AI teams, using Python, SQL, and evaluation tools to scale AI products.

Qualifications

  • Strong Python skills for data manipulation, analysis, and tooling (pandas/NumPy).
  • Solid SQL and data modeling experience with large datasets.

Responsibilities

  • Build and maintain dataset and evaluation infrastructure with ground-truth controls.
  • Diagnose and fix measurement-path issues between offline evals and production accuracy.
  • Own prompt optimization from candidate selection through production rollout and post-launch validation.
  • Automate eval pipelines and scale tooling across multiple AI products.
  • Create reusable tooling for error analysis, audits, and experimentation.
  • Iterate on labeling frameworks and knowledge capture systems.
  • Analyze ontology/schema changes and their propagation across models.

Skills

Python
SQL
LLM accuracy
Eval datasets
Ground-truth labeling
Prompt engineering
Ontology design
Team collaboration

Tools

Braintrust
Langfuse

Job description

Procore Technologies, Inc. in Auckland is seeking a Senior Software Engineer to own the data, tooling, and evaluation infrastructure for AI-powered features.

You will build robust data pipelines, improve model quality, and drive prompts and models from candidate selection through production rollout and validation. You will collaborate with engineering, product, and AI teams, using Python, SQL, and evaluation tools to scale AI products.

Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Senior AI Evaluation Engineer — LLM Quality & Pipelines (Auckland)
Senior AI Evaluation Engineer — LLM Quality & Pipelines (Auckland)

Procore Technologies • New Zealand

Hybrid
NZD 92,000 - 127,000
Equity compensation
Bonus incentive
Senior AI Data Science Engineer for LLM Evaluation
Senior AI Data Science Engineer for LLM Evaluation

Procore • New Zealand

Hybrid
NZD 92,000 - 127,000
Equity compensation
Bonus incentive
Senior AI Data Engineer - LLM Evaluation
Senior AI Data Engineer - LLM Evaluation

DroneDeploy New Zealand Limited • Auckland

On-site
NZD 92,000 - 127,000
Senior AI Evaluation & Data Infra Engineer - Auckland
Senior AI Evaluation & Data Infra Engineer - Auckland

DroneDeploy, Inc. • Auckland

Hybrid
NZD 92,000 - 127,000
Equity compensation
Bonus incentive compensation
Senior LLM Evaluation & Data Quality Engineer
Senior LLM Evaluation & Data Quality Engineer

Procore Technologies • Auckland

On-site
NZD 92,000 - 127,000
Equity compensation
Bonus incentive
Senior Software Engineer, Data Science
Senior Software Engineer, Data Science

DroneDeploy New Zealand Limited • Auckland

On-site
NZD 92,000 - 127,000
Senior Software Engineer, Data Science
Senior Software Engineer, Data Science

Procore Technologies • New Zealand

Hybrid
NZD 92,000 - 127,000
Equity compensation
Bonus incentive
Senior Software Engineer, Data Science
Senior Software Engineer, Data Science

Procore • New Zealand

Hybrid
NZD 92,000 - 127,000
Equity compensation
Bonus incentive
Senior Software Engineer, Data Science
Senior Software Engineer, Data Science

Procore Technologies • Auckland

On-site
NZD 92,000 - 127,000
Equity compensation
Bonus incentive
Senior Software Engineer, Data Science
Senior Software Engineer, Data Science

DroneDeploy, Inc. • Auckland

Hybrid
NZD 92,000 - 127,000
Equity compensation
Bonus incentive compensation
Top locations