Databricks Data Engineer Senior Analyst - Avanade

OpenTalent; City of Brisbane; Hybrid; AUD 150,000 - 210,000; Full time

Job summary

OpenTalent seeks an experienced Databricks Data Engineer to design, develop, and maintain scalable data pipelines on Azure Databricks for advanced analytics. You will collaborate with Data Scientists to deliver clean, structured datasets and ensure governance and security compliant with banking regulations.

The role emphasizes PySpark, Spark optimization, and Azure components (Data Factory, Synapse, Delta Lake) in an Agile setting across Brisbane, Sydney, or Melbourne, with flexible location and

Qualifications

• 2+ years hands-on experience in Azure Databricks.
• Strong proficiency in Apache Spark and PySpark.
• Experience with Azure Data Factory, Synapse, Blob Storage, and Delta Lake.
• Understanding ETL/ELT, data modeling, and data warehousing concepts.
• Proficiency in Python and/or Scala for data processing.
• Familiarity with CI/CD pipelines (Azure DevOps, Git, Terraform).
• Knowledge of data governance, security, and compliance in financial services.

Responsibilities

• Design, develop, and optimize scalable data ingestion pipelines using Azure Data Factory, Databricks, and Spark.
• Build Databricks notebooks with PySpark for data prep and enrichment.
• Collaborate with Data Scientists to deliver clean datasets.
• Implement data quality checks and monitoring.
• Integrate data from structured, semi-structured, and unstructured sources.
• Ensure data security, governance, and regulatory compliance in banking.

Skills

Azure Databricks

PySpark

Python

Scala

Agile working

Education

Bachelor’s or Master’s degree in CS/IS/Engineering

Tools

Azure Data Factory

Azure Synapse

Azure Blob Storage

Delta Lake

Git

Terraform

Azure DevOps

Job description

Role -Databricks Data Engineer Senior Analyst

About the Role

We are seeking experienced Databricks Data Engineers to join our Data & AI. You will play a pivotal role in designing, developing, and maintaining scalable data pipelines and transformation workflows on Azure Databricks, enabling advanced analytics and data engineering initiatives.

Key Responsibilities

• Design, develop, and optimize scalable data ingestion pipelines using Azure Data Factory, Databricks, and Apache Spark.
• Build and maintain Databricks notebooks using PySpark for data preparation, transformation, and enrichment to support Data Science use cases.
• Collaborate closely with Data Scientists to understand data requirements and ensure timely delivery of clean, structured datasets.
• Implement data quality checks, validation rules, and monitoring mechanisms.
• Integrate data from various sources including structured, semi-structured, and unstructured formats.
• Ensure data security, governance, and compliance with banking regulations.
• Optimize Spark jobs for performance and cost-efficiency in Azure.
• Participate in code reviews, design discussions, and agile ceremonies.
• Document data pipelines, workflows, and technical decisions.

Required Skills & Qualifications

Technical Skills

• Experience in data engineering, with at least 2+ years hands-on experience in Azure Databricks.
• Strong proficiency in Apache Spark, especially PySpark for distributed data processing.
• Experience with Azure Data Factory, Azure Synapse, Azure Blob Storage, and Delta Lake.
• Solid understanding of ETL/ELT processes, data modeling, and data warehousing concepts.
• Proficiency in Python and/or Scala for data processing.
• Experience with CI/CD pipelines using tools like Azure DevOps, Git, and Terraform (preferred).
• Familiarity with SQL, NoSQL, and data lake architectures.
• Knowledge of data governance, security, and compliance in financial services.

Soft Skills

• Strong analytical and problem-solving skills.
• Excellent communication and stakeholder management abilities.
• Ability to work collaboratively in cross-functional teams.
• Agile mindset and experience working in Scrum/Agile environments.

Preferred Qualifications

• Bachelor’s or Master’s degree in Computer Science, Information Systems, Engineering, or related field.
• Databricks Certified Data Engineer Associate/Professional certification is a plus.
• Experience working in banking or financial services domain is highly desirable.
• Australian citizenship is required, and candidates must be eligible to obtain security clearance.
• Location is flexible across Brisbane, Sydney, and Melbourne.