Lead Data Engineer

Firmable; Melbourne; Hybrid; AUD 140,000 - 210,000; Full time

Job summary

Firmable Pty Ltd in Melbourne is seeking an experienced data extraction and ETL lead to design end-to-end pipelines, ensure data quality, and own the architecture driving scalable data products.

You will work across sourcing, extraction, LLM integration, observability, and tooling, shaping standard practices, performance, and reliability in production systems for diverse business-critical use cases.

Qualifications

• 6-10 years shipping production extraction, ETL, or data pipeline systems in business-critical environments; deep web extraction at scale with anti-bot defences, proxy architecture, JS rendering, schema drift, and recovery.
• Shipped LLMs inside extraction pipelines as production systems: structured outputs, versioned prompts, labelled eval sets, logged traces, drift detected on vendor model updates. You can show the repo.
• Built agents and tool-calling pipelines: architected agent workflows, written SKILL. MD specs others depend on, run tool-calling systems in production.
• Sharp judgement on rules vs. LLMs: deterministic logic first, defend the call either way, and codify the standard for a team.
• Expert Python: production-grade, performance-aware, concurrency and scale; advanced SQL for complex transformations.
• Extensive Airflow or equivalent: orchestration, dependency management, recovery patterns, cost optimisation at production scale.

Responsibilities

• Own data sourcing and ETL architecture across end-to-end pipelines.
• Develop extraction frameworks and anti-bot strategies for JS-heavy sources.
• Design LLM-based pipelines with structured outputs and validation.
• Ensure observability: prompts, traces, evals, and drift detection.
• Manage tool-calling and agent workflows in production.

Skills

Python

SQL

Airflow

LLM integration

Data pipeline design

Production systems

Automation

Tools

Snowflake

Redshift

AWS

GCP

Job description

The candidate will have responsibilities across the following functions:

Data Sourcing and ETL architecture:

• End-to-end pipeline design: extraction, normalisation, deduplication, validation, and load; plus cost, performance, and reliability of the whole layer.
• Sourcing layer standards: coverage, accuracy, schema design; the rule-vs-LLM standard codified so the team applies it without you.
• Cost and performance optimisation: cost ceilings with the token math behind them, incremental processing, recovery design, scheduling.

Extraction systems and frameworks:

• Extractor framework: patterns, abstractions, and tooling other engineers ship into; new extractors are fast to build and reliable to run.
• Hard-source extractors: anti-bot defences, JS-heavy rendering, schema drift, low- quality structure; plus the proxy and IP rotation strategy behind them.
• Agentic extraction pipelines: rule-based triage, LLM escalation, structured-output validation, retries, human-review queues.

LLM infrastructure and observability:

• LLMs as production systems: versioned prompts, labelled eval sets, measured precision and recall, prompt versioning you can roll back, judges debugged on real data.
• Eval and observability scaffolding: eval frameworks, prompt versioning, traces, drift detection when a vendor silently updates a model.
• Model-choice playbook: cheap models for classification, stronger models for nuanced extraction, frontier models for hard edge cases; revised as model economics shift.

Skills library and orchestration:

• Skills library: versioned SKILL. MD specs for recurring extraction patterns, invocable by engineers and agents alike.
• Orchestration: Airflow or equivalent patterns that scale with data volumes and source counts; dependency management, recovery, cost-aware scheduling.

Requirements:

• 6-10 years shipping production extraction, ETL, or data pipeline systems in business-critical environments; deep web extraction at scale with experience designing around anti-bot defences, proxy architecture, JS rendering, schema drift, and recovery.
• Shipped LLMs inside extraction pipelines as production systems: structured outputs, versioned prompts, labelled eval sets, logged traces, judges debugged on precision/recall, drift detected on vendor model updates. You can show the repo.
• Built agents and tool-calling pipelines: architected agent workflows, written SKILL. MD specs others depend on, run tool-calling systems in production.
• Sharp judgement on rules vs. LLMs: reach for deterministic logic first, defend the call either way, and have codified the standard for a team.
• Expert Python: production-grade, performance-aware, comfortable with concurrency and scale; plus advanced SQL for complex transformations and performance optimisation.
• Extensive Airflow or equivalent: orchestration, dependency management, recovery patterns, cost optimisation at production scale.
• Shipped with agentic IDEs: Claude Code, Cursor, or equivalent as your default mode, with shipped extraction systems to show for it.
• Architecture judgement and a product mindset: when to refactor, optimise, ship, or start again; you care about coverage and accuracy landing with customers, not uptime metrics.
• You live and breathe AI tools. Structured outputs, evals, traces, and LLM tracing.

Highly Valued:

• Cloud data platforms: Snowflake, Redshift; AWS for pipeline deployment (Lambda, S3, ECS, Glue).
• Vector databases, embeddings, or retrieval patterns for matching and deduplication.
• Eval frameworks like Braintrust, Promptfoo, or Inspect; LLM tracing tooling.
• Data quality frameworks with automated testing and anomaly detection at scale.
• B2B data: firmographics, people data, company registries across markets.
• Data privacy and compliance: GDPR, CCPA.
• Startup or scaleup experience where you shipped fast and owned outcomes end to end.