Senior Data Engineer

NorthBay Solutions LLC; Abu Dhabi; On-site; AED 180,000 - 220,000; Full time

Job summary

NorthBay Solutions LLC is seeking an experienced Senior Data Engineer in Abu Dhabi. The ideal candidate will have 8–10+ years in data engineering, focusing on designing and managing Azure-based data platforms.

The role involves building scalable data solutions, optimizing data ingestion frameworks, and implementing security best practices. Expertise in Azure Data Services, data warehousing, and strong programming skills in languages like Python are essential for success in this position.

Qualifications

• 8–10+ years of experience in Data Engineering.
• Strong hands-on experience with Microsoft Azure Data Services.
• Expertise in Azure Data Factory, Azure Synapse Analytics, and Azure Data Lake Storage.

Responsibilities

• Design and maintain scalable data platforms on Microsoft Azure.
• Build batch and real-time data ingestion frameworks.
• Develop ETL/ELT processes using cloud-native practices.

Skills

Azure Data Services

Python

SQL

Data Warehousing

ETL/ELT

PySpark

Data Governance

DevOps

Tools

Azure Data Factory

Azure Synapse Analytics

Databricks

Job description

Location: Onsite – Abu Dhabi, UAE

Employment Type: Full-Time, Permanent

Job Summary

We are seeking an experienced Senior Data Engineer with 8–10+ years of experience in designing, developing, and managing modern data platforms, data pipelines, and cloud-based analytics solutions. The ideal candidate will have strong expertise in Azure Data Services, large-scale data processing, data warehousing, ETL/ELT frameworks, and cloud-native data architectures. The role requires hands‑on experience in building scalable, secure, and high-performance data solutions that support enterprise analytics, reporting, AI, and business intelligence initiatives.

Key Responsibilities

• Design, develop, and maintain scalable data platforms and data pipelines on Microsoft Azure.
• Build and optimize batch and real-time data ingestion frameworks from multiple structured and unstructured data sources.
• Design and implement data lake, data warehouse, and lakehouse architectures to support analytics and reporting workloads.
• Develop and manage ETL/ELT processes using modern cloud-native data engineering practices.
• Implement data transformation, cleansing, validation, and quality frameworks to ensure data accuracy and reliability.
• Collaborate with business stakeholders, data analysts, data scientists, and application teams to understand data requirements and deliver scalable solutions.
• Optimize data storage, processing, and query performance across enterprise data platforms.
• Implement security, governance, monitoring, and compliance best practices across Azure environments.
• Support integration of data platforms with AI/ML, business intelligence, and enterprise applications.
• Participate in architecture reviews, code reviews, troubleshooting, and technical mentoring activities.
• Ensure high availability, scalability, and operational excellence of data platforms and pipelines.

Required Skills & Qualifications

• 8–10+ years of experience in Data Engineering, Data Warehousing, and Enterprise Data Platform development.
• Strong hands‑on experience with Microsoft Azure Data Services.
• Expertise in Azure Data Factory (ADF), Azure Synapse Analytics, Azure Data Lake Storage (ADLS Gen2), and Azure SQL Database.
• Experience building and managing large-scale ETL/ELT pipelines and data integration solutions.
• Strong proficiency in SQL, query optimization, and database performance tuning.
• Hands‑on experience with PySpark, Apache Spark, and distributed data processing frameworks.
• Strong programming skills in Python, Scala, or Java.
• Experience with dimensional modeling, data warehousing concepts, and modern lakehouse architectures.
• Experience working with structured, semi‑structured, and unstructured data.
• Strong understanding of data governance, data quality, metadata management, and security best practices.
• Experience with REST APIs, data integration patterns, and enterprise system connectivity.
• Hands‑on experience with Git, CI/CD pipelines, and DevOps practices.
• Strong analytical, problem‑solving, and communication skills.

Preferred Qualifications

• Experience with Microsoft Fabric, OneLake, Dataflows, and Fabric Data Engineering workloads.
• Experience with Databricks, Delta Lake, and lakehouse implementations.
• Knowledge of real‑time streaming technologies such as Azure Event Hubs, Apache Kafka, or Azure Stream Analytics.
• Experience supporting AI/ML and advanced analytics workloads through enterprise data platforms.
• Familiarity with Power BI datasets, semantic models, and enterprise reporting architectures.
• Experience with data governance tools such as Microsoft Purview.
• Microsoft Azure Data Engineering certifications are highly preferred.
• Experience working in Agile/Scrum environments.

Nice to Have

• Experience with Microsoft Fabric Data Engineering and Analytics solutions.
• Exposure to MLOps and DataOps practices.
• Knowledge of containerization technologies such as Docker and Kubernetes.
• Experience with Infrastructure as Code (Terraform, ARM Templates, or Bicep).
• Familiarity with Snowflake, BigQuery, or other cloud data warehouse platforms.
• Experience with enterprise‑scale data migration and modernization projects.
• Understanding of Generative AI, Vector Databases, and data platforms supporting AI workloads.