Data Engineer

Confidential; Abu Dhabi; On-site; AED 260,000 - 360,000; Full time

Job summary

Confidential is seeking a senior, hands-on Data Engineer to build and scale a modern data platform while directly contributing to implementation. The role balances technical leadership with hands-on delivery and requires staying close to the code and pipelines.

You will mentor engineers, establish engineering standards, review code, and collaborate with architects, analytics and AI teams to drive reliable, scalable data solutions across multi-source environments.

Qualifications

• 8–10+ years of experience in data engineering.
• Hands-on Databricks implementation experience.
• Proficient in Python, SQL and Spark.
• Experience designing scalable ETL/ELT pipelines.
• Experience with data warehouses, lakehouses and data marts.
• Knowledge of metadata‑driven ingestion frameworks.
• Production engineering practices: monitoring, testing, reliability.
• Data governance, metadata, lineage, security.
• Batch and real-time processing experience.
• Cloud platform experience, preferably Azure.

Responsibilities

• Build and scale a modern data platform.
• Provide technical leadership to the engineering team.
• Mentor engineers and set delivery standards.
• Hands-on implementation and code reviews.
• Collaborate with architects, analytics, AI teams and stakeholders.

Skills

Databricks

Python

SQL

Spark

ETL pipelines

Data warehouses

Lakehouse

Mentoring

Data governance

Real-time processing

Production engineering

Stakeholder comms

Tools

Databricks

Azure

Job description

We are looking for a senior, hands‑on Data Engineer who can build and scale a modern data platform while also providing technical leadership to the engineering team. The candidate must still be close to the technology and directly involved in implementation. We are not looking for a profile that is mainly focused on architecture, advisory work or people management.

Core requirements

• 8–10+ years of experience in Data Engineering, ETL development or data-platform engineering, including experience leading or mentoring engineers.
• Strong recent and demonstrable Databricks implementation experience. The candidate should have personally built or implemented solutions on Databricks, rather than only defining the architecture or managing delivery.
• Strong hands‑on capability in Python, SQL and Spark.
• Proven experience designing and building scalable ETL/ELT and ingestion pipelines in complex, multi‑source environments.
• Experience creating structured data layers, data warehouses, data marts and lakehouse solutions for enterprise reporting, analytics and AI use cases.
• Experience with metadata‑driven ingestion frameworks, reusable engineering patterns and platform administration would be highly relevant.
• Strong understanding of production engineering practices, including pipeline monitoring, automated testing, error handling, data freshness, reliability and performance.
• Practical knowledge of data governance, catalogues, metadata, lineage, data quality, MDM, role‑based access control and security.
• Experience with batch and real‑time processing. Exposure to Databricks Structured Streaming or Delta Live Tables would be useful.
• Hands‑on experience with a major cloud platform, preferably Azure, although AWS or GCP experience may also be considered.
• Ability to set engineering standards, review code, mentor engineers and lead delivery without moving away from hands‑on technical work.
• Clear and concise communication, with the ability to work with architects, analytics teams, AI teams and business stakeholders.

Preferred background

• Databricks, Azure or other relevant cloud/data‑engineering certifications would be beneficial.
• Exposure to APIs and data services used by enterprise applications, analytics platforms, ML pipelines or agentic systems would be useful.