Azure Data Engineer: Pipelines, Databricks & Python

HCL Technologies Limited

Sydney

On-site

AUD 120,000 - 150,000

Full time

3 days ago

Be an early applicant

Application generator

A complete application in a minute — tailored resume and cover letter, ready to send.

Get past ATS filters

Job summary

HCL Technologies Limited is seeking a Technical Specialist to oversee Azure Data Factory (ADF), Azure Databricks, SQL and Python data processing. You will design, develop, and maintain data pipelines and ETL workflows on the Azure cloud platform to optimize data transformation and storage.

Responsibilities include building scalable pipelines, writing optimized SQL queries, and developing automation scripts.

Qualifications

Proficiency in Azure Data Factory for data integration and orchestration.
Experience with Azure Databricks for data engineering and analytics.
Strong SQL skills for data querying.
Proficient in Python for data processing and automation.
Strong analytical and problem-solving skills.
Excellent communication and collaboration abilities.
Ability to adapt to changing priorities and manage multiple tasks.

Responsibilities

Design, develop, and implement data pipelines using Azure Data Factory to move data between various sources and data warehouses.
Utilize Azure Databricks for data transformations and analytics to derive valuable insights from data.
Write and optimize SQL queries to extract, manipulate, and analyze data from multiple sources.
Develop and maintain Python scripts for data processing and automation tasks.
Collaborate with cross-functional teams to understand data requirements and ensure data solutions meet business needs.
Monitor data pipeline performance and troubleshoot issues to ensure data accuracy and reliability.
Stay updated on industry trends and best practices related to Azure Data Factory, Azure Databricks, SQL, and Python.