DATA ENGINEER

Duncan & Ross; Abu Dhabi; On-site; AED 100,000 - 150,000; Full time

Job summary

A leading technology firm in Abu Dhabi is seeking a Data Engineer to design and maintain robust data pipelines while ensuring data integrity and security. The ideal candidate will have at least 5 years of experience in data engineering, strong knowledge of Azure Databricks, and proficiency in SQL and Python. This role offers the opportunity to collaborate with data scientists and optimize data workflows in a dynamic environment.

Qualifications

• 5 years of experience in a data engineering or data platform role.
• Strong experience with Azure Databricks, Data Lake Storage, PySpark, and Jupyter Notebooks.
• Proficiency in SQL, Python, and distributed data processing frameworks.
• Understanding of data governance, privacy, and security best practices.

Responsibilities

• Design constructs and maintain data pipelines for data ingestion, transformation, and storage.
• Ensure data integrity, quality, and security throughout the data lifecycle.
• Optimize data processing workflows for performance and scalability.

Skills

Data pipeline design

Azure Databricks

SQL

Python

Data governance

Tools

Data Lake Storage

Azure Data Factory

Jupyter Notebooks

Job description

KEY RESPONSIBILITIES:

• Design construct and maintain data pipelines for data ingestion transformation and storage
• Ensure data integrity quality and security throughout the data lifecycle.
• Optimize data processing workflows for performance and scalability.
• Collaborate with data scientists and analysts to facilitate data accessibility.

QUALIFICATIONS:

• 5 years of experience in a data engineering or data platform role.
• Strong experience with Azure Databricks Data Lake Storage PySpark and Jupyter Notebooks.
• Proficiency in SQL Python and distributed data processing frameworks.
• Experience working with Azure Data Factory Azure Synapse or similar ETL/ELT tools.
• Understanding of data governance privacy and security best practices.
• Familiarity with supporting ML/AI workflows including data preparation for machine learning pipelines.
• Excellent problem-solving and communication skills

Vertical

Technology