Senior Data Engineer

ALOIS Australia

Sydney

On-site

AUD 180,000 - 230,000

Full time

2 hours ago
Be an early applicant

Get a reply from this employer

Your resume and cover letter, tailored to exactly what they’re hiring for.

Get past ATS filters

Job summary

ALOIS Australia is seeking a Senior Data Engineer to lead the design and delivery of secure, scalable data solutions across on-premises and cloud platforms in Sydney.

You will provide technical leadership, designing batch and near-real-time pipelines and integrating data into trusted data assets for reporting, analytics and downstream applications. You will mentor other data engineers and help establish reusable engineering patterns to improve platform reliability and cost efficiency.

Qualifications

  • Seven or more years of experience in data engineering or data warehousing.
  • Strong track record delivering end-to-end data solutions.
  • Experience with SQL, Python, Spark, and Databricks.
  • Experience designing cloud-based analytics or reporting pipelines.
  • Experience with batch and near-real-time processing.
  • Experience with data profiling, ETL/ELT, and source-to-target mappings.
  • Experience with orchestration tools like Airflow or similar.
  • CI/CD, version control, automated testing and release-management practices.
  • Experience with JSON, XML and Avro data formats.
  • Demonstrated ability to lead, mentor engineers and influence technology decisions.
  • Azure data services and cloud data warehouse technologies.
  • Industry certifications in Azure, Databricks or related platforms.

Responsibilities

  • Lead the design and delivery of complex, production-grade data pipelines and data assets across Azure, Databricks and on-premises platforms.
  • Translate product priorities and business requirements into practical engineering solutions.
  • Own or lead ingestion, transformation, modelling, storage and data consumption designs.
  • Ensure solutions meet security, reliability, performance and scalability requirements.
  • Optimise cloud and platform usage to balance performance and costs.
  • Promote reusable data engineering patterns and best practices.
  • Review designs, mappings, test strategies and release plans.
  • Lead data profiling, wrangling and analysis for new datasets.
  • Collaborate with SMEs to ensure data assets meet business needs.
  • Design data products for reporting, analytics and downstream apps.
  • Coordinate planning, governance, architecture and release activities across teams.
  • Ensure pipelines meet SLAs and monitoring/operational requirements.
  • Lead resolution of complex production issues and drive improvements.
  • Provide mentoring and peer reviews for data engineers.
  • Contribute to roadmaps, risk management and continuous improvement.

Skills

Advanced SQL
Python
Data modeling
Problem solving
Technical leadership
Mentoring
Batch processing
Streaming / near-real-time
Data profiling
ETL/ELT design
CI/CD
Version control
Testing

Education

Degree in Computer Science, Mathematics, Statistics or related

Tools

Azure Databricks
Spark
Airflow
Control-M
HDFS
Hive
Impala
Cloudera
JSON / XML / Avro
REST APIs
Kafka / streaming frameworks

Job description

We are looking for a Senior Data Engineer to lead the design and delivery of secure, scalable and cost-effective data solutions across on-premises and cloud platforms.

You will provide technical leadership across complex data engineering initiatives, designing batch and near-real-time pipelines and integrating data into trusted data assets for reporting, analytics and downstream applications. You will also help establish reusable engineering patterns, improve platform reliability and mentor other data engineers.

Key responsibilities
  • Lead the design and delivery of complex, production-grade data pipelines and data assets across Azure, Databricks and on-premises big-data platforms.
  • Translate product priorities, business outcomes and technical requirements into practical engineering solutions.
  • Own or lead solution design for ingestion, transformation, modelling, storage and data consumption.
  • Ensure solutions meet non-functional requirements, including security, reliability, performance, scalability, maintainability and usability.
  • Optimise cloud and platform usage to meet performance requirements while managing compute and operating costs.
  • Establish and promote reusable data engineering patterns, standards and practices.
  • Review technical designs, source-to-target mappings, test strategies, implementation plans and operational documentation.
  • Lead data profiling, wrangling and analysis for new and complex datasets.
  • Work with subject-matter experts to understand business rules and ensure data assets are fit for purpose.
  • Design data products appropriate to different consumption patterns, including reporting, analytics and downstream applications.
  • Coordinate planning, governance, architecture, engineering, testing and release activities across multiple teams.
  • Ensure pipelines meet agreed service levels, dependencies, monitoring and operational requirements.
  • Lead the resolution of complex production issues and drive permanent improvements.
  • Provide technical mentoring, coaching and peer review for data engineers.
  • Contribute to engineering roadmaps, delivery estimates, risk management and continuous improvement.
  • Promote secure, governed and high-quality data practices across the delivery community.
What you will bring
Essential experience and skills
  • Seven or more years of experience in data engineering, data warehousing or a related field.
  • Strong experience designing and delivering end-to-end data engineering solutions.
  • Advanced SQL and Python, with practical experience using Spark or equivalent distributed processing technologies.
  • At least two years of experience designing and delivering solutions with Azure Databricks.
  • Experience delivering cloud-based analytics or reporting pipelines.
  • Strong experience with batch processing and practical exposure to near-real-time or streaming solutions.
  • Experience with data profiling, source-to-target mapping, ETL/ELT development, testing, implementation and SQL optimisation.
  • Experience with orchestration tools such as Airflow, Control-M or equivalent.
  • Experience applying CI/CD, version control, automated testing and release-management practices.
  • Experience designing solutions for structured and unstructured data, including JSON, XML and Avro.
  • Demonstrated ability to provide technical leadership, mentor engineers and make sound engineering decisions.
  • Azure data services and cloud data warehouse technologies.
  • Cloudera, Hive, Impala, HDFS, HBase or Hue.
  • Kafka, Spark Streaming or other streaming frameworks.
  • REST APIs, microservices or Data-as-a-Service architectures.
  • Experience with high-volume batch or near-real-time processing.
  • Experience with financial or management-reporting processes.
  • Relevant Azure, Databricks or other industry certifications.
Qualifications

A degree in Computer Science, Mathematics, Statistics, Information Management, Finance, Economics or a related discipline is desirable. Equivalent practical experience will also be considered.

In this role, you will be expected to:

  • Act as a technical lead for complex initiatives.
  • Review and challenge designs, implementation approaches and engineering decisions.
  • Establish practical and reusable engineering patterns.
  • Identify and manage delivery, operational and technical risks.
  • Support estimation, sequencing and prioritisation of engineering work.
Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Senior Data Engineer
Senior Data Engineer

ALOIS UK • Perth

On-site
AUD 140,000 - 190,000
Data Engineer
Data Engineer

XPT Software • Sydney

On-site
AUD 140,000 - 190,000
Senior Data Engineer
Senior Data Engineer

Xpt Software Australia Pty • New South Wales

On-site
AUD 140,000 - 180,000
Senior Data Engineer
Senior Data Engineer

DX1 • Melbourne

On-site
AUD 160,000 - 230,000
Senior Data Engineer
Senior Data Engineer

Work Initiatives • North Sydney

On-site
AUD 150,000 - 210,000
Senior Data and Analytics Engineer
Senior Data and Analytics Engineer

SG Fleet • Sydney

On-site
AUD 150,000 - 210,000
Wellness days
Parental leave
Learning budget
Senior Data Engineer
Senior Data Engineer

Elysium Digital • Melbourne

On-site
AUD 120,000 - 180,000
Data Engineer
Data Engineer

XPT Software Australia Pty Ltd • Sydney

On-site
AUD 180,000 - 230,000
Azure Data Engineer
Azure Data Engineer

Tech Mahindra Limited • Sydney

On-site
AUD 130,000 - 180,000
Senior Data Engineer
Senior Data Engineer

Elysium Digital • Sydney

On-site
AUD 130,000 - 180,000
Top locations