Senior Data Engineer (Databricks, Lakehouse & AI-BI)

NR Consulting; Abu Dhabi; On-site; AED 300,000 - 520,000; Full time

Job summary

NR Consulting in Abu Dhabi seeks a Senior Data Engineer to own end-to-end data pipelines and lakehouse development using Databricks Lakehouse, Lakeflow, and Unity Catalog. You will design scalable Delta Lake tables, implement CDC-based processing, and enable Genie AI/BI dashboards with semantic layers.

The role emphasizes governance, CI/CD practices, and collaboration with analytics stakeholders to deliver trusted data assets in a fast-paced environment.

Qualifications

• Strong hands-on Databricks expertise with PySpark and SQL.

Responsibilities

• Pipeline orchestration using Databricks Lakeflow and Azure Data Factory.

Skills

Databricks

PySpark

SQL

Workflows

Delta Lake

Unity Catalog

Lakeflow

CDC

Data quality

CI/CD

Git

Databricks Repos

Tools

Azure Data Factory

Job description

Title: Senior Data Engineer (Databricks Lakehouse & AI-BI)

Location: ABU DHABI

Exp: 6 Years

Job Description:

Key Responsibilities

1. 1. Pipeline Orchestration: Design and build robust data ingestion and orchestration pipelines with Databricks (Lakeflow Jobs/Workflows) as the primary platform and Azure Data Factory supporting legacy/external needs taking full ownership of pipeline reliability and the transition toward Databricks-native orchestration.
2. 2. Lakehouse Data Engineering: Build and maintain scalable Delta Lake tables under Unity Catalog schema design access controls lineage tracking and data quality across bronze/silver/gold layers.
3. 3. Lakeflow Pipeline Development: Build declarative ETL pipelines using Databricks Lakeflow incorporating CDC incremental processing and data quality validations.
4. 4. AI/BI & Genie Enablement: Design and maintain semantic layers metrics and certified datasets to support Genie spaces and AI/BI dashboards enabling natural-language querying and self-service analytics.
5. 5. SharePoint/External Source Integration: Configure and maintain Lakeflow Connect (or similar) integrations for ingesting SharePoint and other external data sources into the Lakehouse with minimal manual effort.
6. 6. Governance & Best Practices: Apply Unity Catalog governance controls access policies data masking audit logging and uphold strong engineering practices (CI/CD version control testing).

Must-Have Skills

• Strong hands-on Databricks expertise PySpark/SQL Workflows/Jobs orchestration Delta Lake Unity Catalog
• Working knowledge of Azure Data Factory
• Experience with Lakeflow (Declarative Pipelines/Connect) including CDC and data quality validations
• Exposure to Databricks AI/BI or Genie (semantic layers certified datasets)
• Familiarity with CI/CD and version control (Git Databricks Repos/Asset Bundles)
• Demonstrated ability to own deliverables fully and resolve issues independently
• Strong communication skills for working with business/analytics stakeholders

Required Experience:

Senior IC