Staff Data Scientist - GenAI

Cover Genius; Sydney; On-site; AUD 210,000 - 320,000; Full time

Job summary

Cover Genius, a global embedded protection platform in Sydney, seeks a Staff Data Scientist for Generative AI on the Customer Ops Team. You will own the design and delivery of agentic AI systems, from idea to production, tackling high-impact problems across claims, operations and partner products.

You will lead architecture and evaluation, ship core components, and raise AI capability across teams, collaborating with engineering, product, and analytics to ensure accuracy, safety, and scalable

Qualifications

• 8+ years in data science or machine learning, including at least 2 to 3 years building and shipping LLM-based or agentic systems into production.
• Deep expertise in LLM evaluation: ground-truth dataset and rubric design, LLM-as-judge methods and how to validate them, statistical reasoning with small samples, and error analysis of multi-step or multi-agent systems.
• Strong Python engineering, with practical experience of agent and tool-use patterns (e.g. MCP, function calling), workflow orchestration (e.g. Temporal or similar), and a major cloud AI platform (e.g. AWS Bedrock).
• A track record of influencing technical direction beyond your own team: shaping architecture, influencing engineering standards, and mentoring engineers who were new to AI/ML.
• Bachelor's degree in a relevant field. A postgraduate degree and/or record of academic achievement is also desirable.

Responsibilities

• Own the design of agentic AI systems. Decide where models add value and where deterministic code should do the work, define agent boundaries, tools and contracts, and design the controls that determine when a system may act on its own.
• Build the evaluation foundation. Create ground-truth datasets, labelling rubrics with measured inter-rater agreement, and eval harnesses that gate every prompt and model change, measuring both component-level and end-to-end performance against a defined production bar.
• Ship production systems hands-on. Build prompts, agents, tool integrations, evaluation pipelines and LLM-as-judge scorers, and lead the error analysis that turns failures into specific, prioritised fixes.
• Lead engineers through influence. Run design reviews, pair on implementation, and establish the patterns teams use to build, test and release AI components, so engineers can independently ship eval-gated changes.
• Raise AI capability across the organisation. Turn each engagement into reusable guidance, templates, shared tooling and hands-on sessions that other engineering and product teams can adopt.
• Own the outcomes, not just the models. Define and track the metrics that matter for each system, such as automation rate, decision accuracy, unsafe-action rate and cost per task, and report to leadership with a clear account of what is moving them and why.
• Shape where we invest next. Assess new opportunities for agentic AI across the business, recommend what to build, what to buy and what not to automate, and run fast, falsifiable proofs of concept to decide.

Skills

LLM evaluation

Python engineering

Agentic AI

Production deployment

Leadership without authority

Education

Bachelor's in CS/Related

Postgrad desirable

Tools

AWS Bedrock

Temporal

MCP

Function calling

Job description

The Company

Cover Genius is the global infrastructure for embedded protection. Active in over 60 countries and all 50 US States, we protect the customers of the world's largest digital companies, including Klarna, Revolut, Stripe, Priceline, Agoda, Booking.com, Turkish Airlines, Tongcheng Travel, eBay, and Uber, with seamless, end-to-end experiences. Cover Genius has protected more than 73M customers globally across 240M policies with USD $3.2BN in gross written sales.

Cover Genius is the global infrastructure for embedded protection. Active in over 60 countries and all 50 US States, we protect the customers of the world's largest digital companies, including Klarna, Revolut, Stripe, Priceline, Agoda, Booking.com, Turkish Airlines, Tongcheng Travel, eBay, and Uber, with seamless, end-to-end experiences. Cover Genius has protected more than 73M customers globally across 240M policies with USD $3.2BN in gross written sales.
Coming off a stellar year with 40% YoY revenue growth and a recent $100M capital raise, putting our valuation at $1.9BN, we are accelerating into our next phase of growth. As part of our team, you'll help drive our AI-first roadmap, developing hyper-personalization engines, agentic distribution, and automated claims infrastructure, while building the scalable technology powering the fast-growing $70B embedded protection market.

Our people are:

Accountable, customer-obsessed, collaborative, driven

Our people are not:

Passive, defensive, siloed, hesitant

About The Role

As a Staff Data Scientist for Generative AI on our Customer Ops Team, you will own the design and delivery of agentic AI systems that take on complex, judgement-heavy work across the business, from automated claims decisions to customer operations and partner-facing/customer-facing products. You will go where the highest-impact problem is, take it from idea to production, and leave behind systems and teams that can keep improving without you. To drive success in this role, you will have deep hands-on experience taking LLM-based and agentic systems from prototype to production, a rigorous approach to evaluation, and the credibility to shape engineering decisions without formal authority. As the Staff Data Scientist, you will set the architecture and evaluation strategy for our agentic systems, build and ship core components yourself, and raise the AI engineering capability of the teams you work with. Regular collaboration with Engineering, Product, Operations, Compliance, Analytics team will be key in ensuring that the systems we build are accurate, auditable and safe to scale.

NOTE

We are actively considering candidates at both Senior and Staff levels for this role. Final leveling and compensation will be determined based on overall experience and interview performance, so we encourage you to apply regardless of which title best fits you today!

Key Responsibilities

• Own the design of agentic AI systems. Decide where models add value and where deterministic code should do the work, define agent boundaries, tools and contracts, and design the controls that determine when a system may act on its own.
• Build the evaluation foundation. Create ground-truth datasets, labelling rubrics with measured inter-rater agreement, and eval harnesses that gate every prompt and model change, measuring both component-level and end-to-end performance against a defined production bar.
• Ship production systems hands-on. Build prompts, agents, tool integrations, evaluation pipelines and LLM-as-judge scorers, and lead the error analysis that turns failures into specific, prioritised fixes.
• Lead engineers through influence. Run design reviews, pair on implementation, and establish the patterns teams use to build, test and release AI components, so engineers can independently ship eval-gated changes.
• Raise AI capability across the organisation. Turn each engagement into reusable guidance, templates, shared tooling and hands-on sessions that other engineering and product teams can adopt.
• Own the outcomes, not just the models. Define and track the metrics that matter for each system, such as automation rate, decision accuracy, unsafe-action rate and cost per task, and report to leadership with a clear account of what is moving them and why.
• Shape where we invest next. Assess new opportunities for agentic AI across the business, recommend what to build, what to buy and what not to automate, and run fast, falsifiable proofs of concept to decide.

What You Will Bring

• 8+ years in data science or machine learning, including at least 2 to 3 years building and shipping LLM-based or agentic systems into production, where you owned the outcome after launch and not only the prototype.
• Deep expertise in LLM evaluation: ground-truth dataset and rubric design, LLM-as-judge methods and how to validate them, statistical reasoning with small samples, and error analysis of multi-step or multi-agent systems.
• Strong Python engineering, with practical experience of agent and tool-use patterns (e.g. MCP, function calling), workflow orchestration (e.g. Temporal or similar), and a major cloud AI platform (e.g. AWS Bedrock).
• A track record of influencing technical direction beyond your own team: shaping architecture, influencing engineering standards, and mentoring engineers who were new to AI/ML.
• Bachelor's degree in a relevant field. A postgraduate degree and/or record of academic achievement is also desirable.

Proficiencies And Attributes

• Leadership without authority. You bring data scientists and engineers along by making them better, through clear reasoning, pairing and patient explanation rather than title.
• Judgement about when not to use AI. You are as comfortable arguing that a component should be plain code as you are designing an agent.
• Range and fast ramp-up. You can walk into an unfamiliar domain, learn how the work is really done from the people who do it, and turn that into a system design quickly.
• Clarity with non-technical audiences. You can explain to an operator, a compliance lead or an