Senior LLM Evaluation & Data Quality Engineer

Procore Technologies

Auckland

On-site

NZD 92,000 - 127,000

Full time

14 days+

Get a reply from this employer

Your resume and cover letter, tailored to exactly what they’re hiring for.

Get past ATS filters

Benefits offered by this job

Equity compensation
Bonus incentive

Job summary

Procore Technologies in Auckland is seeking a Senior Software Engineer for its Data Science team to own the accuracy of LLM-powered features that convert drone and robotics data into actionable insights.

You will collaborate with engineering, product, and AI teams to build data, tooling, and evaluation infrastructure, using Python, SQL, and LLM evaluation to improve product quality and reliability. This role reports to the Director of Engineering and is based in our Auckland office.

Qualifications

  • Strong Python and SQL experience with data manipulation and tooling.
  • Background in data science with focus on LLM accuracy and evaluation.
  • Experience building evaluation datasets and error analysis tools.
  • Familiarity with ground-truth labeling and data quality checks.
  • Knowledge of prompt engineering and ontology/schema design.

Responsibilities

  • Build and maintain dataset and evaluation infrastructure with quality controls.
  • Diagnose gaps between offline evals and production accuracy.
  • Own prompt optimization across lifecycle from candidate selection to production.
  • Automate eval pipelines and scalable tooling for multiple AI products.
  • Create internal tooling for error analysis and experiments.
  • Iterate on labeling frameworks and schema propagation analyses.
  • Prioritize work by likely impact and align with teams.

Skills

Python
SQL
LLM evaluation
Data analysis
Prompt engineering
Ontology design
Team collaboration

Tools

Braintrust
Langfuse

Job description

Procore Technologies in Auckland is seeking a Senior Software Engineer for its Data Science team to own the accuracy of LLM-powered features that convert drone and robotics data into actionable insights.

You will collaborate with engineering, product, and AI teams to build data, tooling, and evaluation infrastructure, using Python, SQL, and LLM evaluation to improve product quality and reliability. This role reports to the Director of Engineering and is based in our Auckland office.

Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Senior AI Data Science Engineer for LLM Evaluation
Senior AI Data Science Engineer for LLM Evaluation

Procore • New Zealand

Hybrid
NZD 92,000 - 127,000
Equity compensation
Bonus incentive
Senior AI Evaluation Engineer — LLM Quality & Pipelines (Auckland)
Senior AI Evaluation Engineer — LLM Quality & Pipelines (Auckland)

Procore Technologies • New Zealand

Hybrid
NZD 92,000 - 127,000
Equity compensation
Bonus incentive
Senior AI Data Engineer - LLM Evaluation
Senior AI Data Engineer - LLM Evaluation

DroneDeploy New Zealand Limited • Auckland

On-site
NZD 92,000 - 127,000
Senior AI Engineer – LLM Evaluation & Data Pipelines
Senior AI Engineer – LLM Evaluation & Data Pipelines

Procore Technologies, Inc. • Auckland

On-site
NZD 92,000 - 127,000
Equity compensation
Bonus incentive compensation
Senior Software Engineer, Data Science
Senior Software Engineer, Data Science

DroneDeploy New Zealand Limited • Auckland

On-site
NZD 92,000 - 127,000
Senior Software Engineer, Data Science
Senior Software Engineer, Data Science

Procore Technologies • New Zealand

Hybrid
NZD 92,000 - 127,000
Equity compensation
Bonus incentive
Senior Software Engineer, Data Science
Senior Software Engineer, Data Science

Procore Technologies • Auckland

On-site
NZD 92,000 - 127,000
Equity compensation
Bonus incentive
Senior Software Engineer, Data Science
Senior Software Engineer, Data Science

Procore • New Zealand

Hybrid
NZD 92,000 - 127,000
Equity compensation
Bonus incentive
Senior Software Engineer, Data Science
Senior Software Engineer, Data Science

DroneDeploy, Inc. • Auckland

Hybrid
NZD 92,000 - 127,000
Equity compensation
Bonus incentive compensation
Senior Software Engineer, Data Science
Senior Software Engineer, Data Science

Procore Technologies, Inc. • Auckland

On-site
NZD 92,000 - 127,000
Equity compensation
Bonus incentive compensation
Top locations