Senior Azure Databricks Data Engineer — AI Lakehouse Lead

Efforts Solutions IT LLC; Abu Dhabi; Hybrid; AED 250,000 - 450,000; Full time

Benefits offered by this job

Health Insurance

Visa

Job summary

Efforts Solutions IT LLC is seeking a Senior Data Engineer (Azure Databricks) to lead the design and delivery of an enterprise-grade AI-powered agentic lakehouse on Azure.

The role is remote initially, with potential relocation to Abu Dhabi for onsite or hybrid engagement. You will build ingestion pipelines, govern data with Unity Catalog, and enable AI-driven data workflows across Medallion layers.

Qualifications

• 7+ years in data engineering leading lakehouse builds.
• Proven track record defining target lakehouse architectures.
• Hands-on Azure Databricks: Delta Lake, Unity Catalog, DLT/Lakeflow, Workflows.
• Strong PySpark and SQL; data modelling across Medallion layers.
• Production ingestion/transformation pipelines with automated data-quality checks.
• Azure data services: ADLS Gen2, Data Factory, Key Vault, private networking.
• Experience with ERP data cleansing or migration.
• Practical use of LLMs in data workflows (DQ, metadata, text-to-SQL).
• Git and CI/CD for data pipelines.

Responsibilities

• Lead the lakehouse architecture: ingestion, Bronze/Silver/Gold layering, Delta Lake standards.
• Build ingestion pipelines from ERP databases, apps, SharePoint and APIs.
• Set up Unity Catalog for governance: catalogs, lineage, access control.
• Design AI agents for data profiling, quality rules and anomaly detection.
• Expose governed data to AI agents via SQL endpoints and vector indexes.
• Embed data-quality checks with quarantine and human review for fixes.
• Set engineering standards and review internal/external teams.
• Apply cost controls: cluster policies and tagging.

Skills

Data leadership

Lakehouse architecture

Data modeling

Data governance

PySpark

SQL

AI integration

Communication

Tools

Azure Databricks

Delta Lake

Unity Catalog

DLT / Lakeflow

Workflows

ADLS Gen2

Data Factory

Key Vault

Git & CI/CD

Azure OpenAI

Azure AI

Private networking

Job description

Efforts Solutions IT LLC is seeking a Senior Data Engineer (Azure Databricks) to lead the design and delivery of an enterprise-grade AI-powered agentic lakehouse on Azure.

The role is remote initially, with potential relocation to Abu Dhabi for onsite or hybrid engagement. You will build ingestion pipelines, govern data with Unity Catalog, and enable AI-driven data workflows across Medallion layers.