Senior Data Engineer

Airswift; Abu Dhabi; On-site; AED 300,000 - 420,000; Full time

Job summary

Airswift is seeking a Senior Data Engineer for on-site work in Abu Dhabi, UAE. The role focuses on building and migrating data pipelines to Databricks, optimizing PySpark transformations, and aligning with CI/CD IaC practices within Azure-heavy environments.

The candidate should bring proven Databricks production experience, strong coding skills, and the ability to operate in pod-based delivery teams. Excellent English communication is required.

Qualifications

• Proven production-level Databricks experience.
• Strong hands-on coding capability.
• Experience building and optimizing modern data pipelines.
• Ability to work within pod-based delivery teams.
• Experience owning pipeline migration and rebuild activities from design through validation.
• Databricks Databricks hands-on experience.
• Delta Live Tables.
• Unity Catalog.
• Workflows.
• Production experience required (not lab-based experience).
• Data Engineering & Development PySpark.
• Advanced SQL.
• Data modelling.
• Schema design.
• Medallion architecture.
• Lakehouse architecture.
• DevOps & Deployment Modern CI/CD for data pipelines.
• Infrastructure as Code (IaC).
• Git-based deployment.
• Communication Strong written and verbal English skills.
• Ability to interact directly with Databricks Professional Services (DBX).

Responsibilities

• Implement Databricks pipelines using Delta Live Tables.
• Rewrite Foundry transforms into optimised PySpark on Databricks.
• Move from Foundry to modern CI/CD with IaC-managed Databricks workspaces.
• Tune Databricks Photon performance to match Foundry interactivity.
• Contribute to reconciliation engine between Palantir and Databricks outputs.
• Manage pipeline rebuild, validation, internal testing and UAT prep.

Skills

Databricks

PySpark

Delta Live Tables

Unity Catalog

CI/CD

IaC

SQL

Data modelling

GIT

Python

Airflow

Palantir Foundry

Code migration

Tools

Databricks

Delta Live Tables

Unity Catalog

dbt

Airflow

Terraform

Git

Infra as Code

Job description

Senior Data Engineer - UAE

Position Overview

Location: Abu Dhabi, UAE

Work Requirement: Client requirement to have people onsite with office visits.

Key Responsibilities

• Implementing Databricks pipelines using Delta Live Tables (declarative pipelines).
• Rewriting proprietary Foundry transforms into optimised PySpark on Databricks.
• Moving from Foundry's build system to modern CI/CD (IaC-managed Databricks workspaces).
• Performing performance tuning on Databricks Photon to match or exceed Foundry interactivity.
• Contributing to the reconciliation engine (Palantir vs Databricks output equivalence).
• Managing pipeline rebuild, validation, internal testing, and UAT preparation.

Required Skills & Experience

• Proven production-level Databricks experience.
• Strong hands-on coding capability.
• Experience building and optimizing modern data pipelines.
• Ability to work within pod-based delivery teams.
• Experience owning pipeline migration and rebuild activities from design through validation.
• Databricks Databricks hands-on experience.
• Delta Live Tables.
• Unity Catalog.
• Workflows.
• Production experience required (not lab-based experience).
• Data Engineering & Development PySpark.
• Advanced SQL.
• Data modelling.
• Schema design.
• Medallion architecture.
• Lakehouse architecture.
• DevOps & Deployment Modern CI/CD for data pipelines.
• Infrastructure as Code (IaC).
• Git-based deployment.
• Communication Strong written and verbal English skills.
• Ability to interact directly with Databricks Professional Services (DBX).

Preferred Qualifications

• Certifications Databricks Certified Data Engineer Professional.
• Associate certification at minimum.
• Domain Experience Financial services.
• Audit.
• Asset management.
• Platform Experience Prior Palantir Foundry exposure, including: Code Repo.
• Code Workbook.
• Pipeline Builder.
• Migration Experience.
• Experience with migrations from legacy platforms such as: Informatica.
• Hyperion.
• Oracle.
• Palantir Foundry.
• Databricks.

Skills

• Candidates with any of the following experience will be advantageous: Azure Data Factory (ADF).
• Azure Synapse.
• dbt.
• Airflow.
• Equivalent orchestration tools.

Note: Client estate is Azure-heavy.