Senior Data Engineer — Real-Time Lakehouse on AWS

CoStar Group, Inc.; Sydney; On-site; AUD 170,000 - 210,000; Full time

Benefits offered by this job

Extra annual leave

Leadership development programs

LinkedIn Learning

Perkbox discounts

Job summary

CoStar Group is seeking a Senior Data Engineer to build data platform capabilities for batch and streaming pipelines in the APAC region. You will design, develop, and operate data assets that support analytics, ML/AI, and data-driven decisions across product teams.

You will implement data solutions on AWS, Snowflake, and Databricks, optimise performance, and ensure cost-efficiency while collaborating with data scientists, engineers, and product managers.

Qualifications

• 5+ years of hands-on commercial data/software engineering experience.
• 4+ years of SQL and Python development experience.
• Experience with data integration/orchestration tools: Airflow, DBT, AWS Glue.
• Experience designing data solutions on AWS, Snowflake, Databricks and Lakehouse.
• Experience with semi-structured/unstructured data and data modelling in agile environments.
• Hands-on experience building streaming pipelines with Kafka and Flink.
• Knowledge of IaC tools: CDK, CloudFormation, or Terraform.
• Degree in computer science, engineering, mathematics, statistics, or related field.

Responsibilities

• Design and build reusable, scalable data solutions on a Lakehouse platform.
• Develop robust pipelines using Airflow, DBT, Snowflake, Databricks and AWS services.
• Create real-time data pipelines with Kafka and Flink for low-latency use cases.
• Connect to diverse data sources via APIs, SaaS, or streaming.
• Architect data solutions on AWS, focusing on cost-efficiency and best practices.
• Model data with modern techniques to support analytics across Prod & Data teams.
• Collaborate with data scientists, analysts, and engineers to deliver value.
• Proactively identify opportunities to improve data processes and reduce costs.

Skills

SQL

Python

Data modeling

Cloud native

Education

Degree in computer science or related technical field

Tools

Apache Airflow

DBT

AWS Glue

Snowflake

Databricks

Apache Kafka

Apache Flink

CDK/CloudFormation/Terraform

Job description

CoStar Group is seeking a Senior Data Engineer to build data platform capabilities for batch and streaming pipelines in the APAC region. You will design, develop, and operate data assets that support analytics, ML/AI, and data-driven decisions across product teams.

You will implement data solutions on AWS, Snowflake, and Databricks, optimise performance, and ensure cost-efficiency while collaborating with data scientists, engineers, and product managers.