Databricks Data Engineer Consultant

Avanade Inc.; City of Melbourne; On-site; AUD 120,000 - 180,000; Full time

Job summary

Avanade Inc. is seeking an experienced Databricks Data Engineer Consultant to design, develop, and maintain scalable data pipelines on Azure Databricks, enabling advanced analytics and data engineering initiatives.

You will collaborate with Data Scientists to understand data requirements, implement data quality checks, and optimize Spark jobs for performance and cost on Azure, while upholding data security and governance standards in regulated banking contexts.

Responsibilities

• Design, develop, and optimize scalable data ingestion pipelines using Azure Data Factory, Databricks, and Apache Spark.
• Build and maintain Databricks notebooks using PySpark for data preparation and transformation.
• Collaborate closely with Data Scientists to understand data requirements and ensure timely delivery of clean, structured datasets.
• Implement data quality checks, validation rules, and monitoring mechanisms.
• Integrate data from various sources including structured, semi-structured, and unstructured formats.
• Ensure data security, governance, and compliance with banking regulations.
• Optimize Spark jobs for performance and cost-efficiency in Azure.
• Participate in code reviews, design discussions, and agile ceremonies.
• Document data pipelines, workflows, and technical decisions.

Skills

Databricks

PySpark

Azure Data Factory

Apache Spark

Tools

Azure Databricks

Azure Data Factory

Job description

Role -Databricks Data Engineer Consultant

About the Role

We are seeking experienced Databricks Data Engineers to join our Data & AI. You will play a pivotal role in designing, developing, and maintaining scalable data pipelines and transformation workflows on Azure Databricks, enabling advanced analytics and data engineering initiatives.

Key Responsibilities

• Design, develop, and optimize scalable data ingestion pipelines using Azure Data Factory, Databricks, and Apache Spark.
• Build and maintain Databricks notebooks using PySpark for data preparation, transformation, and enrichment to support Data Science use cases.
• Collaborate closely with Data Scientists to understand data requirements and ensure timely delivery of clean, structured datasets.
• Implement data quality checks, validation rules, and monitoring mechanisms.
• Integrate data from various sources including structured, semi-structured, and unstructured formats.
• Ensure data security, governance, and compliance with banking regulations.
• Optimize Spark jobs for performance and cost-efficiency in Azure.
• Participate in code reviews, design discussions, and agile ceremonies.
• Document data pipelines, workflows, and technical decisions.

Equal Employment Opportunity Statement

All employment decisions shall be made without regard to age, race, creed, color, religion, sex, national origin, ancestry, disability status, veteran status, sexual orientation, gender identity or expression, genetic information, marital status, citizenship status or any other basis as protected by federal, state, or local law.

Job candidates will not be obligated to disclose sealed or expunged records of conviction or arrest as part of the hiring process.

Avanade is committed to providing veteran employment opportunities to our service men and women.

Please read Avanade's Hiring and Privacy Statement for more information on how we process your data during the Recruiting and Hiring process.