Senior Data Engineer: Lakehouse, Streaming & Cloud

CoStar Group; Sydney; Hybrid; AUD 140,000 - 190,000; Full time

Benefits offered by this job

Extra annual leave days

Leadership development programs

LinkedIn Learning

Perkbox discounts

Job summary

CoStar Group is seeking a Senior Data Engineer for Domain to build and operate data platforms and pipelines across the APAC region. You will design scalable data solutions on a Lakehouse, leveraging Snowflake, Databricks and AWS services to power data-driven decisions.

You will collaborate with data scientists, analysts and product teams to deliver reliable, cost-efficient data assets and real-time streaming capabilities using Kafka and Flink.

Qualifications

• 5+ years in software/data engineering.
• 4+ years SQL and Python development.
• Experience with data integration and orchestration tools: Airflow, DBT, AWS Glue.
• Proficiency with Snowflake, Databricks and Lakehouse architectures.
• Experience on AWS cloud infrastructure.
• Experience with semi-structured/unstructured data.
• Strong data modelling in agile environments.
• Hands-on streaming pipelines with Kafka and Flink.
• IaC tools: CDK, CloudFormation, or Terraform.
• Degree in CS/Engineering/Math/Statistics or related field.

Responsibilities

• Design and build reusable, scalable data solutions on Lakehouse platforms.
• Develop robust pipelines using DBT, Airflow, Snowflake, Databricks and AWS services.
• Create real-time data pipelines with Kafka and Flink for low-latency use cases.
• Integrate data from SaaS, APIs, and streaming sources across cloud architecture.
• Architect data solutions on AWS, Snowflake and Databricks for cost-efficiency.
• Optimize Lakehouse table formats and manage schema evolution and partitioning.
• Apply modern data modelling techniques in an agile setting.
• Write optimized SQL and Spark workloads for performance and cost control.
• Collaborate with data science, analytics, product and engineering teams.
• Proactively identify opportunities to improve data processes and reliability.

Skills

SQL

Python

Airflow

DBT

Snowflake

Databricks

AWS

Kafka

Flink

Lakehouse

IaC (CDK/TF/CFN)

Education

Bachelor's degree in Computer Science or related field

Tools

Airflow

DBT

Snowflake

Databricks

AWS Glue

Kafka

Flink

Terraform/CloudFormation/CDK

Job description

CoStar Group is seeking a Senior Data Engineer for Domain to build and operate data platforms and pipelines across the APAC region. You will design scalable data solutions on a Lakehouse, leveraging Snowflake, Databricks and AWS services to power data-driven decisions.

You will collaborate with data scientists, analysts and product teams to deliver reliable, cost-efficient data assets and real-time streaming capabilities using Kafka and Flink.