Senior AI Evaluation Engineer — LLM Quality & Pipelines (Auckland)

Procore Technologies

New Zealand

Hybrid

NZD 92,000 - 127,000

Full time

14 days+

A complete application in a minute

Tailored resume and cover letter, ready to send.

Get past ATS filters

Benefits offered by this job

Equity compensation
Bonus incentive

Job summary

Procore Technologies is seeking a Senior Software Engineer for its Data Science team in Auckland. You will own LLM feature accuracy, build evaluation infra, and drive production-ready prompts and models for drone/robotics data insights.

You will collaborate with engineering, product, and AI teams to create data tooling, evaluation pipelines, and scalable QA processes. This role reports to the Director of Engineering and starts immediately.

Qualifications

  • Proven experience with Python for data manipulation and tooling.
  • Strong SQL and data modeling skills.
  • Background in data science with knowledge of LLM accuracy, including visual reasoning.
  • Experience building evaluation datasets and conducting error analysis.
  • Experience designing ground-truth labeling systems and measuring data quality.
  • Familiar with prompt engineering and ontology/schema design.
  • Collaborative, ownership-driven mindset and cross-functional teamwork.
  • Auckland location with at least 2 days per week in the office.

Responsibilities

  • Build and maintain dataset and evaluation infrastructure with ground-truth quality controls.
  • Diagnose and fix measurement-path issues between offline evals and production accuracy.
  • Own prompt optimization across the lifecycle from selection to production rollout.
  • Automate eval pipelines and scalable tooling for multiple AI products.
  • Create internal tooling for error analysis, audits, and experimentation.
  • Scale labeling frameworks and knowledge capture systems.
  • Analyze ontology/schema changes and their impact across models.
  • Prioritize work by likely impact.

Skills

Python
SQL
LLM evaluation
Eval datasets
Ground-truth labeling
Prompt engineering
Ontology design
Team collaboration
Domain knowledge

Tools

Braintrust
Langfuse

Job description

Procore Technologies is seeking a Senior Software Engineer for its Data Science team in Auckland. You will own LLM feature accuracy, build evaluation infra, and drive production-ready prompts and models for drone/robotics data insights.

You will collaborate with engineering, product, and AI teams to create data tooling, evaluation pipelines, and scalable QA processes. This role reports to the Director of Engineering and starts immediately.

Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Senior AI Engineer – LLM Evaluation & Data Pipelines
Senior AI Engineer – LLM Evaluation & Data Pipelines

Procore Technologies, Inc. • Auckland

On-site
NZD 92,000 - 127,000
Equity compensation
Bonus incentive compensation
Senior AI Data Science Engineer for LLM Evaluation
Senior AI Data Science Engineer for LLM Evaluation

Procore • New Zealand

Hybrid
NZD 92,000 - 127,000
Equity compensation
Bonus incentive
Senior AI Data Engineer - LLM Evaluation
Senior AI Data Engineer - LLM Evaluation

DroneDeploy New Zealand Limited • Auckland

On-site
NZD 92,000 - 127,000
Senior LLM Evaluation & Data Quality Engineer
Senior LLM Evaluation & Data Quality Engineer

Procore Technologies • Auckland

On-site
NZD 92,000 - 127,000
Equity compensation
Bonus incentive
Senior AI Evaluation & Data Infra Engineer - Auckland
Senior AI Evaluation & Data Infra Engineer - Auckland

DroneDeploy, Inc. • Auckland

Hybrid
NZD 92,000 - 127,000
Equity compensation
Bonus incentive compensation
Senior Software Engineer, Data Science
Senior Software Engineer, Data Science

DroneDeploy New Zealand Limited • Auckland

On-site
NZD 92,000 - 127,000
Senior Software Engineer, Data Science
Senior Software Engineer, Data Science

Procore Technologies • New Zealand

Hybrid
NZD 92,000 - 127,000
Equity compensation
Bonus incentive
Senior Software Engineer, Data Science
Senior Software Engineer, Data Science

Procore • New Zealand

Hybrid
NZD 92,000 - 127,000
Equity compensation
Bonus incentive
Senior Software Engineer, Data Science
Senior Software Engineer, Data Science

Procore Technologies • Auckland

On-site
NZD 92,000 - 127,000
Equity compensation
Bonus incentive
Senior Software Engineer, Data Science
Senior Software Engineer, Data Science

DroneDeploy, Inc. • Auckland

Hybrid
NZD 92,000 - 127,000
Equity compensation
Bonus incentive compensation
Top locations