Lead Data Engineer: Production ETL & LLM Pipelines

Firmable Pty Ltd; Melbourne; Hybrid; AUD 140,000 - 210,000; Full time

Job summary

Firmable Pty Ltd in Melbourne is seeking an experienced data extraction and ETL lead to design end-to-end pipelines, ensure data quality, and own the architecture driving scalable data products.

You will work across sourcing, extraction, LLM integration, observability, and tooling, shaping standard practices, performance, and reliability in production systems for diverse business-critical use cases.

Qualifications

• 6-10 years shipping production extraction, ETL, or data pipeline systems in business-critical environments; deep web extraction at scale with anti-bot defences, proxy architecture, JS rendering, schema drift, and recovery.
• Shipped LLMs inside extraction pipelines as production systems: structured outputs, versioned prompts, labelled eval sets, logged traces, drift detected on vendor model updates. You can show the repo.
• Built agents and tool-calling pipelines: architected agent workflows, written SKILL. MD specs others depend on, run tool-calling systems in production.
• Sharp judgement on rules vs. LLMs: deterministic logic first, defend the call either way, and codify the standard for a team.
• Expert Python: production-grade, performance-aware, concurrency and scale; advanced SQL for complex transformations.
• Extensive Airflow or equivalent: orchestration, dependency management, recovery patterns, cost optimisation at production scale.

Responsibilities

• Own data sourcing and ETL architecture across end-to-end pipelines.
• Develop extraction frameworks and anti-bot strategies for JS-heavy sources.
• Design LLM-based pipelines with structured outputs and validation.
• Ensure observability: prompts, traces, evals, and drift detection.
• Manage tool-calling and agent workflows in production.

Skills

Python

SQL

Airflow

LLM integration

Data pipeline design

Production systems

Automation

Tools

Snowflake

Redshift

AWS

GCP

Job description

Firmable Pty Ltd in Melbourne is seeking an experienced data extraction and ETL lead to design end-to-end pipelines, ensure data quality, and own the architecture driving scalable data products.

You will work across sourcing, extraction, LLM integration, observability, and tooling, shaping standard practices, performance, and reliability in production systems for diverse business-critical use cases.