Staff Generative AI Scientist: Agentic Systems Lead

Cover Genius; Sydney; On-site; AUD 210,000 - 320,000; Full time

Job summary

Cover Genius, a global embedded protection platform in Sydney, seeks a Staff Data Scientist for Generative AI on the Customer Ops Team. You will own the design and delivery of agentic AI systems, from idea to production, tackling high-impact problems across claims, operations and partner products.

You will lead architecture and evaluation, ship core components, and raise AI capability across teams, collaborating with engineering, product, and analytics to ensure accuracy, safety, and scalable

Qualifications

• 8+ years in data science or machine learning, including at least 2 to 3 years building and shipping LLM-based or agentic systems into production.
• Deep expertise in LLM evaluation: ground-truth dataset and rubric design, LLM-as-judge methods and how to validate them, statistical reasoning with small samples, and error analysis of multi-step or multi-agent systems.
• Strong Python engineering, with practical experience of agent and tool-use patterns (e.g. MCP, function calling), workflow orchestration (e.g. Temporal or similar), and a major cloud AI platform (e.g. AWS Bedrock).
• A track record of influencing technical direction beyond your own team: shaping architecture, influencing engineering standards, and mentoring engineers who were new to AI/ML.
• Bachelor's degree in a relevant field. A postgraduate degree and/or record of academic achievement is also desirable.

Responsibilities

• Own the design of agentic AI systems. Decide where models add value and where deterministic code should do the work, define agent boundaries, tools and contracts, and design the controls that determine when a system may act on its own.
• Build the evaluation foundation. Create ground-truth datasets, labelling rubrics with measured inter-rater agreement, and eval harnesses that gate every prompt and model change, measuring both component-level and end-to-end performance against a defined production bar.
• Ship production systems hands-on. Build prompts, agents, tool integrations, evaluation pipelines and LLM-as-judge scorers, and lead the error analysis that turns failures into specific, prioritised fixes.
• Lead engineers through influence. Run design reviews, pair on implementation, and establish the patterns teams use to build, test and release AI components, so engineers can independently ship eval-gated changes.
• Raise AI capability across the organisation. Turn each engagement into reusable guidance, templates, shared tooling and hands-on sessions that other engineering and product teams can adopt.
• Own the outcomes, not just the models. Define and track the metrics that matter for each system, such as automation rate, decision accuracy, unsafe-action rate and cost per task, and report to leadership with a clear account of what is moving them and why.
• Shape where we invest next. Assess new opportunities for agentic AI across the business, recommend what to build, what to buy and what not to automate, and run fast, falsifiable proofs of concept to decide.

Skills

LLM evaluation

Python engineering

Agentic AI

Production deployment

Leadership without authority

Education

Bachelor's in CS/Related

Postgrad desirable

Tools

AWS Bedrock

Temporal

MCP

Function calling

Job description

Cover Genius, a global embedded protection platform in Sydney, seeks a Staff Data Scientist for Generative AI on the Customer Ops Team. You will own the design and delivery of agentic AI systems, from idea to production, tackling high-impact problems across claims, operations and partner products.

You will lead architecture and evaluation, ship core components, and raise AI capability across teams, collaborating with engineering, product, and analytics to ensure accuracy, safety, and scalable