Senior Ai Data Engineer Azure Databricks and Agentic Lakehouse

Efforts Solutions IT LLC

Abu Dhabi

Hybrid

AED 250,000 - 450,000

Full time

29 hours ago

Be an early applicant

Get a reply from this employer

Your resume and cover letter, tailored to exactly what they’re hiring for.

Get past ATS filters

Benefits offered by this job

Health Insurance

Visa

Job summary

Efforts Solutions IT LLC is seeking a Senior Data Engineer (Azure Databricks) to lead the design and delivery of an enterprise-grade AI-powered agentic lakehouse on Azure.

The role is remote initially, with potential relocation to Abu Dhabi for onsite or hybrid engagement. You will build ingestion pipelines, govern data with Unity Catalog, and enable AI-driven data workflows across Medallion layers.

Qualifications

7+ years in data engineering leading lakehouse builds.
Proven track record defining target lakehouse architectures.
Hands-on Azure Databricks: Delta Lake, Unity Catalog, DLT/Lakeflow, Workflows.
Strong PySpark and SQL; data modelling across Medallion layers.
Production ingestion/transformation pipelines with automated data-quality checks.
Azure data services: ADLS Gen2, Data Factory, Key Vault, private networking.
Experience with ERP data cleansing or migration.
Practical use of LLMs in data workflows (DQ, metadata, text-to-SQL).
Git and CI/CD for data pipelines.

Responsibilities

Lead the lakehouse architecture: ingestion, Bronze/Silver/Gold layering, Delta Lake standards.
Build ingestion pipelines from ERP databases, apps, SharePoint and APIs.
Set up Unity Catalog for governance: catalogs, lineage, access control.
Design AI agents for data profiling, quality rules and anomaly detection.
Expose governed data to AI agents via SQL endpoints and vector indexes.
Embed data-quality checks with quarantine and human review for fixes.
Set engineering standards and review internal/external teams.
Apply cost controls: cluster policies and tagging.

Skills

Data leadership

Lakehouse architecture

Data modeling

Data governance

PySpark

SQL

AI integration

Communication

Tools

Azure Databricks

Delta Lake

Unity Catalog

DLT / Lakeflow

Workflows

ADLS Gen2

Data Factory

Key Vault

Git & CI/CD

Azure OpenAI

Azure AI

Private networking

Job description

Job Description

Company: Efforts Solutions IT

Job Location: Remote Initially | Abu Dhabi, UAE (Potential Onsite Relocation)

Experience: 7+ Years

Reporting To: AI & Data Manager

About The Opportunity

Efforts Solutions IT is looking for a highly skilled Senior Data Engineer (Azure Databricks) to lead the architecture, development, and implementation of an enterprise-grade AI-powered Agentic Lakehouse for a major enterprise initiative.

This is a hands‑on technical leadership role focused on building a modern data platform using Azure Databricks, Delta Lake, Unity Catalog, PySpark, and Azure Data Factory, integrating enterprise data with intelligent AI‑driven automation.

The successful candidate will work closely with the AI & Data Manager, internal engineering teams, and external delivery partners to transform complex enterprise data into secure, governed, and reusable data products.

Work Arrangement & Relocation

Initial Engagement: Remote for the first 2 months.
Performance Evaluation: Technical capabilities and delivery performance will be assessed during the initial engagement.
Potential Relocation: Following successful evaluation, the role may require relocation to Abu Dhabi, UAE, for an onsite or hybrid engagement.
Contract Extension: Subject to project requirements, performance, and mutual agreement.

Important: Candidates must be willing and able to relocate to Abu Dhabi after the initial remote engagement if required. Please apply only if you are comfortable with this arrangement.

About The Role

We are building an agentic lakehouse: a Databricks Medallion platform on Azure where AI agents help capture, profile, clean and onboard enterprise data into governed, reusable data products. You will lead its technical design and delivery, taking data from legacy operational systems to a trusted, well‑governed layer.

Key responsibilities

Lead the lakehouse architecture: ingestion, Bronze / Silver / Gold layering, Delta Lake standards and data contracts.
Build ingestion pipelines from ERP databases, business applications, SharePoint and APIs using Databricks and Azure Data Factory.
Set up Unity Catalog for governance: catalogs, lineage, access control and data classification.
Design AI agents that work on the data: schema inference, data‑quality rule generation, anomaly detection and entity resolution.
Expose governed data to AI agents through SQL endpoints, vector indexes and MCP‑style data services.
Embed data‑quality checks with quarantine and human review for low‑confidence fixes.
Set engineering standards and review the work of internal engineers and external partners.
Apply cost controls: cluster policies, compute choices and tagging.

Work scope

Assess current data sources, pipelines and data quality.
Design and implement the lakehouse foundations: Medallion layers, Unity Catalog, ingestion framework and CI/CD for data jobs.
Ingest and cleanse priority business domains from Bronze to Silver.
Prototype AI agents for data profiling, quality rules and onboarding.

Required Qualifications

7+ years in data engineering, including leading lakehouse or data-platform builds.
Proven track record defining target lakehouse architectures and data engineering standards.
Deep hands‑on Azure Databricks: Delta Lake, Unity Catalog (access models, lineage), DLT / Lakeflow, Workflows.
Strong PySpark and SQL; data modelling across Medallion layers.
Hands‑on experience building production ingestion and transformation pipelines with automated data‑quality checks.
Azure data services: ADLS Gen2, Data Factory, Key Vault, private networking.
Experience with data cleansing or migration for ERP systems.
Practical use of LLMs in data workflows (data profiling, DQ rule generation, metadata, text‑to‑SQL).
Git and CI/CD for data pipelines.

Preferred