Lead Data Engineer

Jobtailor; Australia; On-site; AUD 140,000 - 210,000; Full time

Benefits offered by this job

Restaurant d'entreprise

Indemnités de stage/alternance

Job summary

Jobtailor is seeking a Senior Data Engineer in Australia to lead the design and delivery of a greenfield, real-time event-streaming platform using Kafka, Flink, and ClickHouse for segmentation at scale.

You will own end-to-end data pipelines from ingestion and transformation to analytics-ready datasets in BigQuery and your team will set quality, testing, and observability standards while driving cost and latency improvements.

Qualifications

• Strong, hands‑on data engineering fundamentals: pipelines, modelling, correctness, scale, and failure modes.
• Strong SQL and Python, used daily in production.
• Experience designing, building, and scaling production data pipelines end to end.
• Depth in the modern batch stack: dbt, an orchestrator, and a cloud warehouse.
• Data modelling skills for analytics and product use cases at scale.
• Experience with event data at billions‑of‑rows scale.
• Experience with AWS or GCP and infrastructure-as-code.
• Data quality and observability mindset, including testing, monitoring, and alerting.
• Track record of setting technical direction through architecture decisions and mentoring senior engineers.
• Ability to explain data trade-offs to non-technical stakeholders.
• Strong ownership and pragmatic judgment.
• Comfort with ambiguity and shifting priorities.
• Bonus: production streaming systems experience with Kafka and Flink, ideally with ClickHouse.
• Bonus: gaming, adtech, or martech event-data background.
• Bonus: using AI to multiply output and improve team AI fluency.

Responsibilities

• Build and scale real-time event-streaming platforms using Kafka, Flink, and ClickHouse.
• Own data lifecycle from ingestion to delivery and collaborate with stakeholders.
• Mentor engineers and raise engineering standards.

Skills

Data pipelines

SQL

Python

Batch processing

dbt

BigQuery / Cloud warehouse

Data modelling

Event data at scale

AWS / GCP

Infrastructure-as-Code

Data quality & observability

Architecture decisions

Mentoring

Ambiguity tolerance

Kafka & Flink

ClickHouse

AI in workflows

Tools

AWS

GCP

ClickHouse

Orchestrator

Infrastructure as Code

Job description

• Build a greenfield real-time event-streaming platform using Kafka, Flink, and ClickHouse for segmentation and targeting at scale


• Set the technical direction for the platform as products scale


• Own warehouse datasets and transformations end to end using BigQuery, dbt, SQL, and Python


• Establish data quality, testing, and observability standards across streaming and batch


• Tune systems for cost and latency as data volumes scale


• Raise engineering standards through code review, mentoring, and reusable team patterns


• Incorporate AI into personal workflows and the platform, including coding agents and automated quality checks


• Support Immutable Audience's data lifecycle from ingestion and transformation through modelling and delivery to internal and external customers

Requirements

• Strong, hands‑on data engineering fundamentals: pipelines, modelling, correctness, scale, and failure modes


• Strong SQL and Python, used daily in production


• Experience designing, building, and scaling production data pipelines end to end


• Depth in the modern batch stack: dbt, an orchestrator, and a cloud warehouse


• Data modelling skills for analytics and product use cases at scale


• Experience with event data at billions‑of‑rows scale


• Experience with AWS or GCP and infrastructure-as-code


• Data quality and observability mindset, including testing, monitoring, and alerting


• Track record of setting technical direction through architecture decisions and mentoring senior engineers


• Ability to explain data trade-offs to non-technical stakeholders


• Strong ownership and pragmatic judgment


• Comfort with ambiguity and shifting priorities


• Bonus: production streaming systems experience with Kafka and Flink, ideally with ClickHouse


• Bonus: gaming, adtech, or martech event-data background


• Bonus: using AI to multiply output and improve team AI fluency

Core Competencies

Demonstrates expertise in building and scaling real-time event-streaming platforms using Kafka, Flink, and ClickHouse, while ensuring data quality and observability. Proficient in SQL and Python for data engineering, with a strong focus on data lifecycle management and technical direction.

Highest-signal resume keywords

• Kafka Event Streaming


• Flink Stream Processing


• SQL Data Engineering


• Python Programming


• Data Quality and Observability

ATS Optimization Keywords

Hard Skills

• Data Pipeline Design


• Data Modelling


• BigQuery


• Dbt


• Cloud Warehouse


• Event Data Management


• Infrastructure-as-Code


• Testing and Monitoring


• Data Transformation


• Cost and Latency Optimization

Soft Skills

• Ownership


• Pragmatic Judgment


• Communication with Non-Technical Stakeholders


• Mentoring


• Comfort with Ambiguity

Industry Keywords

• Gaming


• Adtech


• Martech


• Event Data


• Data Lifecycle

Tools & Technologies

• AWS


• GCP


• ClickHouse


• Orchestrator


• AI Integration