Senior Data Engineer: Lakehouse & Real-Time Pipelines

CoStar Realty Information

Sydney

On-site

AUD 140,000 - 170,000

Full time

3 days ago
Be an early applicant

A complete application in a minute

Tailored resume and cover letter, ready to send.

Get past ATS filters

Benefits offered by this job

Equity / stock options

Job summary

CoStar Realty Information seeks a Senior Data Engineer to design and build scalable data solutions on a Lakehouse platform, partnering with data scientists, analysts, product managers and engineers across the Domain Group. You will craft robust pipelines using DBT, Airflow, Snowflake, Databricks and AWS, and develop real-time streaming architectures with Kafka and Flink in an AWS cloud environment.

The role emphasizes modern data modelling, cost-efficient cloud data warehousing, and governance

Qualifications

  • 5+ years of hands-on commercial experience in software or data engineering.
  • 4+ years of strong development experience in SQL and Python.
  • Proven experience with data integration and orchestration tools, including Apache Airflow, DBT, and AWS Glue.
  • Expert on modern cloud data platforms and architectures, such as Snowflake, Databricks, and Lakehouse architecture
  • Experience designing and building data solutions on AWS cloud infrastructure.
  • Experience working with semi-structured and unstructured data.
  • Strong data modelling skills with the ability to evolve models in an agile environment.
  • Hands‑on experience building streaming and real-time pipelines with Apache Kafka and Apache Flink.
  • Knowledge of infrastructure-as-code tools such as CDK, CloudFormation, or Terraform.
  • Degree in computer science, engineering, mathematics, statistics, or a related technical field

Responsibilities

  • Design and build reusable, scalable data solutions on a Lakehouse platform.
  • Develop robust data pipelines using DBT, Airflow, Snowflake, Databricks and AWS services.
  • Create real-time data pipelines with Kafka and Flink for low-latency use cases.
  • Integrate data from SaaS, APIs, and streaming sources on cloud architecture.
  • Architect data solutions on AWS, Snowflake and Databricks with cost efficiency.
  • Build and optimise Lakehouse table formats (Iceberg), including partitioning and schema evolution.
  • Model data in alignment with evolving business needs in an agile environment.
  • Write and optimise SQL and Spark workloads for performance and cost efficiency.

Skills

Data engineering
SQL
Python
Data modelling
Streaming pipelines
Cloud architecture
Infrastructure as code

Education

Degree in CS/Engineering/Math/Stats

Tools

Apache Airflow
DBT
AWS Glue
Snowflake
Databricks
Kafka
Flink
CDK
CloudFormation
Terraform

Job description

CoStar Realty Information seeks a Senior Data Engineer to design and build scalable data solutions on a Lakehouse platform, partnering with data scientists, analysts, product managers and engineers across the Domain Group. You will craft robust pipelines using DBT, Airflow, Snowflake, Databricks and AWS, and develop real-time streaming architectures with Kafka and Flink in an AWS cloud environment.

The role emphasizes modern data modelling, cost-efficient cloud data warehousing, and governance

Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Senior Data Engineer: Lakehouse & Real-Time Pipelines
Senior Data Engineer: Lakehouse & Real-Time Pipelines

Domain • Sydney

On-site
AUD 140,000 - 180,000
Extra days of annual leave
Leadership development programs
LinkedIn Learning
+1
Senior Data Engineer — Real-Time Lakehouse on AWS
Senior Data Engineer — Real-Time Lakehouse on AWS

CoStar Group, Inc. • Sydney

On-site
AUD 170,000 - 210,000
Extra annual leave
Leadership development programs
LinkedIn Learning
+1
Senior Data Engineer: Lakehouse, Streaming & Cloud
Senior Data Engineer: Lakehouse, Streaming & Cloud

CoStar Group • Sydney

Hybrid
AUD 140,000 - 190,000
Extra annual leave days
Leadership development programs
LinkedIn Learning
+1
Senior Data Engineer — Lakehouse & Real-Time Platform
Senior Data Engineer — Lakehouse & Real-Time Platform

CoStar Group • New South Wales

On-site
AUD 150,000 - 190,000
Extra days of annual leave
Leadership development programs
LinkedIn Learning
+1
Senior Data Engineer: Cloud Data Lakes & Real-Time Pipelines
Senior Data Engineer: Cloud Data Lakes & Real-Time Pipelines

Work Initiatives • North Sydney

On-site
AUD 150,000 - 210,000
Senior Data Engineer - Product
Senior Data Engineer - Product

CoStar Realty Information • Sydney

On-site
AUD 140,000 - 170,000
Equity / stock options
Data Engineer - Cloud Data Pipelines & Lakehouse
Data Engineer - Cloud Data Pipelines & Lakehouse

Compare The Market • Brisbane

On-site
AUD 110,000 - 170,000
Annual celebrations
Flexible work options
ME leave
+5
Senior Data Engineer: Lakehouse Architect for Global Impact
Senior Data Engineer: Lakehouse Architect for Global Impact

Vivanti Consulting • Sydney

On-site
AUD 120,000 - 180,000
Mentoring Program
360° Progress Reviews
Guilds & Regular Training
+4
Senior Data Engineer: Real-Time Data Pipelines & Lakes
Senior Data Engineer: Real-Time Data Pipelines & Lakes

Elysium Digital • Sydney

On-site
AUD 130,000 - 180,000
Data Engineer - Azure Lakehouse & Pipelines
Data Engineer - Azure Lakehouse & Pipelines

Work Initiatives • Brisbane

On-site
AUD 120,000 - 160,000
Learning & Development programs
LinkedIn Learning access
Staff discounts
+2
Top locations