Data Engineer - Azure / Databricks

Ingrity; Sydney; On-site; AUD 110,000 - 160,000; Full time

Job summary

INGRITY is seeking a hands-on Data Engineer in Sydney to design and maintain scalable data pipelines on Azure Databricks and Data Factory. You will deliver PySpark-based solutions, implement ETL/ELT workflows, and support Lakehouse architectures alongside a team of data professionals.

The role emphasizes collaboration with architects and stakeholders, strong data quality standards and a focus on secure, well-documented engineering practices.

Qualifications

• 3–6 years’ experience in Data Engineering or a similar hands-on data role.
• Commercial experience with Azure Databricks.
• Strong experience with Azure Data Factory (ADF).
• Hands-on capability with Python, PySpark, Spark SQL and SQL.
• Experience building ETL/ELT data pipelines.
• Understanding of Lakehouse and Medallion Architecture.
• Experience with Azure data services (ADLS, Azure SQL, Blob Storage, Key Vault).
• Familiarity with Git, Azure DevOps and CI/CD pipelines.
• Understanding of data modelling, data quality and performance optimisation.
• Strong troubleshooting and problem solving; clear communication with stakeholders.

Responsibilities

• Design, develop and maintain scalable data pipelines using Azure Databricks and Azure Data Factory.
• Build and optimise ETL/ELT workflows across cloud platforms.
• Develop solutions using PySpark, Spark SQL, Python and SQL.
• Work with Databricks notebooks, jobs, clusters and workflows.
• Support Lakehouse and Medallion Architecture patterns.
• Collaborate with Data Engineers, Architects and business stakeholders.
• Maintain data quality, security, documentation and engineering best practices.

Skills

Azure Databricks

Azure Data Factory

Python

PySpark

Spark SQL

SQL

Data pipelines

Git

Azure DevOps

CI/CD

Data quality

Troubleshooting

Communication

Tools

dbt

Terraform

Job description

We’re looking for a Data Engineer with strong Azure and Databricks experience to join a growing data team in Sydney.

This is a hands‑on engineering role where you’ll work across modern cloud data platforms, building and supporting scalable data pipelines and helping deliver reliable, high‑quality data solutions.

In this role, you’ll be

• Designing, developing and maintaining scalable data pipelines using Azure Databricks and Azure Data Factory.
• Building and optimising ETL/ELT workflows across cloud‑based data platforms.
• Developing solutions using PySpark, Spark SQL, Python and SQL.
• Working with Databricks notebooks, jobs, clusters and workflows.
• Supporting Lakehouse and Medallion Architecture patterns across data platforms.
• Working with Azure services including ADLS/Blob Storage, Azure SQL, Key Vault and Azure DevOps.
• Monitoring data pipelines, troubleshooting issues and performing root‑cause and performance analysis.
• Supporting CI/CD, Git‑based development and automated deployments.
• Working closely with Data Engineers, Architects and business stakeholders to deliver reliable data solutions.
• Maintaining strong standards around data quality, security, documentation and engineering best practice.

What we need from you

• Around 3–6 years’ experience in Data Engineering or a similar hands‑on data role.
• Commercial experience working with Azure Databricks.
• Strong experience with Azure Data Factory (ADF).
• Good hands‑on capability with Python, PySpark, Spark SQL and SQL.
• Experience building and maintaining ETL/ELT data pipelines.
• Understanding of Lakehouse and Medallion Architecture.
• Experience working with Azure data services such as ADLS, Azure SQL, Blob Storage and Key Vault.
• Familiarity with Git, Azure DevOps and CI/CD pipelines.
• Understanding of data modelling, data quality and performance optimisation.
• Strong troubleshooting and problem‑solving skills.
• Clear communication skills and the ability to work effectively with both technical and business stakeholders.

Nice to have

• Experience with dbt.
• Exposure to Terraform / Infrastructure as Code.
• Understanding of Databricks Unity Catalog.
• Experience with monitoring tools such as Azure Log Analytics or Splunk.
• Exposure to enterprise‑scale data migration or modernisation programs.

Why INGRITY?

At INGRITY, we’re passionate about helping organisations get more value from their data. You’ll have the opportunity to work alongside experienced data professionals, contribute to interesting client projects and continue developing your skills across modern Azure and Databricks technologies.

If you’re a hands‑on Data Engineer looking for your next opportunity in Sydney, we’d love to hear from you.