Senior AI Data Science Engineer for LLM Evaluation

Procore

New Zealand

Hybrid

NZD 92,000 - 127,000

Full time

14 days+

Get a reply from this employer

Your resume and cover letter, tailored to exactly what they’re hiring for.

Get past ATS filters

Benefits offered by this job

Equity compensation
Bonus incentive

Job summary

Procore is seeking a Senior Software Engineer for its Data Science team in Auckland. You will own the accuracy of LLM-powered features, build data, tooling, and evaluation infra, and raise AI product quality across platforms.

You’ll work with engineering, product, and AI teams to manage data pipelines, ground-truthing, and prompt lifecycle optimization, with a focus on measurable improvements in model performance. Occasional travel and hybrid office expectations apply.

Qualifications

  • Strong Python for data manipulation, analysis, and tooling.
  • Solid SQL and database experience with large datasets.
  • Data science background with knowledge of LLM accuracy.
  • Experience building evaluation datasets and debugging evals with Braintrust or Langfuse.
  • Experience designing ground-truth and labeling systems.
  • Familiarity with prompt engineering and ontology/schema design.
  • Collaborative, ownership-driven mindset.
  • Construction or industrial inspection domain knowledge is a plus.

Responsibilities

  • Build and maintain robust dataset and evaluation infrastructure, including ground-truth quality controls.
  • Diagnose and fix measurement-path issues between offline evals and production accuracy.
  • Own prompt optimization across its full lifecycle, from candidate selection through production rollout and post-launch validation.
  • Automate eval pipelines and build tooling to run them at scale across multiple AI products.
  • Create reusable internal tooling for error analysis, audits, and experimentation.
  • Iterate on and scale existing labeling frameworks and knowledge capture systems.
  • Analyze how changes to ontology and schema propagate across models and infrastructure.
  • Prioritize work by likely impact.

Skills

Python
SQL
Data science
LLM evaluation
Ground-truth labeling
Prompt engineering
Ontology/schema design
Collaboration

Tools

Braintrust
Langfuse
pandas
NumPy

Job description

Procore is seeking a Senior Software Engineer for its Data Science team in Auckland. You will own the accuracy of LLM-powered features, build data, tooling, and evaluation infra, and raise AI product quality across platforms.

You’ll work with engineering, product, and AI teams to manage data pipelines, ground-truthing, and prompt lifecycle optimization, with a focus on measurable improvements in model performance. Occasional travel and hybrid office expectations apply.

Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Senior AI Data Engineer - LLM Evaluation
Senior AI Data Engineer - LLM Evaluation

DroneDeploy New Zealand Limited • Auckland

On-site
NZD 92,000 - 127,000
Senior AI Evaluation Engineer — LLM Quality & Pipelines (Auckland)
Senior AI Evaluation Engineer — LLM Quality & Pipelines (Auckland)

Procore Technologies • New Zealand

Hybrid
NZD 92,000 - 127,000
Equity compensation
Bonus incentive
Senior AI Engineer – LLM Evaluation & Data Pipelines
Senior AI Engineer – LLM Evaluation & Data Pipelines

Procore Technologies, Inc. • Auckland

On-site
NZD 92,000 - 127,000
Equity compensation
Bonus incentive compensation
Senior LLM Evaluation & Data Quality Engineer
Senior LLM Evaluation & Data Quality Engineer

Procore Technologies • Auckland

On-site
NZD 92,000 - 127,000
Equity compensation
Bonus incentive
Senior AI Evaluation & Data Infra Engineer - Auckland
Senior AI Evaluation & Data Infra Engineer - Auckland

DroneDeploy, Inc. • Auckland

Hybrid
NZD 92,000 - 127,000
Equity compensation
Bonus incentive compensation
Senior Software Engineer, Data Science
Senior Software Engineer, Data Science

DroneDeploy New Zealand Limited • Auckland

On-site
NZD 92,000 - 127,000
Senior Software Engineer, Data Science
Senior Software Engineer, Data Science

Procore Technologies • New Zealand

Hybrid
NZD 92,000 - 127,000
Equity compensation
Bonus incentive
Senior Software Engineer, Data Science
Senior Software Engineer, Data Science

Procore Technologies • Auckland

On-site
NZD 92,000 - 127,000
Equity compensation
Bonus incentive
Senior Software Engineer, Data Science
Senior Software Engineer, Data Science

Procore • New Zealand

Hybrid
NZD 92,000 - 127,000
Equity compensation
Bonus incentive
Senior Software Engineer, Data Science
Senior Software Engineer, Data Science

DroneDeploy, Inc. • Auckland

Hybrid
NZD 92,000 - 127,000
Equity compensation
Bonus incentive compensation
Top locations