Senior Data & BI Engineer

DataFirst Solutions; Dubai; On-site; AED 420,000 - 780,000; Full time

Job summary

DataFirst Solutions is seeking a lead Databricks Architect to drive end-to-end delivery of an Azure Databricks Lakehouse for hospitality analytics. You will own architecture, technical decisions, and delivery planning, initiating the build-out and mentoring the broader team.

The role requires hands-on Databricks expertise, Delta Lake mastery, and Power BI semantic modeling. You will collaborate with stakeholders across Revenue Management, Operations, and Finance to ensure successful

Qualifications

• 8–12 years of hands-on Databricks experience across end-to-end implementations.
• Deep Delta Lake knowledge including MERGE, OPTIMIZE, VACUUM, schema evolution.
• Proven ability to design lakehouse architecture with medallion layers.
• Power BI development and semantic modeling experience required.
• Strong data quality, reconciliation, and observability practices.

Responsibilities

• Lead discovery workshops with hospitality business/tech stakeholders to align objectives and outline scope and milestones.
• Define target-state lakehouse architecture and a phased delivery roadmap for reservations, stays, POS, loyalty, and digital domains.
• Create end-to-end implementation strategy covering architecture, design standards, data onboarding, governance, and operating model.
• Define coding standards, branching strategy, CI/CD, and environment setups (dev/test/prod) for the project.
• Design and implement Azure Databricks Lakehouse using Delta Lake and medallion layers; ensure performance and cost optimization.
• Implement and review data ingestion patterns, batch and streaming as needed, from APIs, DBs, files, and event streams.
• Develop scalable transformation frameworks with PySpark / Spark SQL; optimize performance and costs.
• Establish monitoring, alerting, job orchestration, dependency management, and recovery strategies.

Skills

Databricks

Delta Lake

Lakehouse architecture

Power BI

Data quality & observability

Tools

Azure Databricks

Unity Catalog

Azure DevOps

GitHub

Terraform

Job description

Experience: 8–12 years (with strong Databricks depth)

Employment Type: Contract - Full time

We are looking for a lead, hands-on Databricks Lead / Lakehouse Architect who can drive project kickoff, define the implementation strategy, and lead end-to-end delivery of an Azure Databricks Lakehouse along with Microsoft Fabric Power BI semantic modeling and reporting for hospitality analytics. This role will own architecture, technical decisions, delivery planning, stakeholder alignment, and initial build-out, and then help ramp the broader team for implementation and rollout.

Key Responsibilities

• Lead discovery workshops, clarify business objectives, define scope, milestones, and delivery approach.
• Lead discovery workshops with hospitality business/tech stakeholders (Revenue Management, Operations, Finance, Loyalty, Digital).
• Define target-state lakehouse architecture and delivery roadmap (phased onboarding of domains: reservations, stays, POS, loyalty, digital).
• Create end-to-end implementation strategy: architecture, design standards, data onboarding plan, governance, and operating model.
• Define best practices, coding standards, branching strategy, CI/CD approach, and environments (dev/test/prod).
• Design and implement Azure Databricks Lakehouse architecture using Delta Lake and Medallion (Bronze/Silver/Gold).
• Design data ingestion patterns for batch (and streaming if needed) from varied sources (APIs, DBs, files, event streams).
• Build scalable transformation frameworks using PySpark / Spark SQL with performance and cost optimization.
• Implement and review complex pipelines: incremental loads, CDC patterns, SCD Type 1/2, data quality rules, and reconciliation.
• Own performance tuning: partitioning, file sizing, Z-ORDER, caching, adaptive query execution, cluster sizing, autoscaling.
• Implement monitoring/alerting, job orchestration, dependency management, and restart/recovery strategies.

Security, Governance & Operations

• Define security model: RBAC/ACLs, service principals, Key Vault integration, secrets management.
• Implement Unity Catalog strategy (if applicable): catalogs, schemas, lineage, permissions, data sharing.
• Establish operational readiness: runbooks, SLAs, logging, audit, cost governance.
• Act as primary technical point of contact for client stakeholders.
• Produce design documentation (HLD/LLD), implementation roadmap, and knowledge transfer plan.
• Mentor engineers, conduct code reviews, and ensure delivery quality.

Must-Have Skills

• Deep hands-on Databricks experience in end-to-end implementations (not only development).
• Strong experience with Delta Lake (MERGE, OPTIMIZE, VACUUM, schema evolution, time travel).
• Experience designing lakehouse architecture (medallion layers, curated zones, semantic datasets).
• Power BI development experience is mandatory.
• Strong understanding of:
• Data quality frameworks, reconciliation, and observability

Nice-to-Have (Strong Plus)

• Unity Catalog implementation experience (access control, lineage, governance).
• CI/CD for Databricks: Azure DevOps/GitHub, Databricks Repos, asset bundles, Terraform.
• Exposure to BI semantic layers (Microsoft Fabric Power BI)
• Databricks certifications (Data Engineer Associate/Professional).
• AI/Copilot Integration experience is plus