Data Developer

Airswift; Abu Dhabi; On-site; AED 220,000 - 360,000; Part time

Job summary

Airswift is seeking a Data Developer (Databricks) to support the final phase of a Data Platform Modernisation Programme in Abu Dhabi, UAE. The role is onsite (5 days per week) with ASAP start and an end date of 31 October.

You will deliver remaining technical activities, standardise engineering practices, and prepare the platform for long-term operational support. The consultant will focus on migrating from legacy technologies, developing scalable data pipelines, and ensuring operational

Qualifications

• Minimum 5 years hands-on Databricks development experience.
• Experience with Delta Live Tables, Spark Declarative Pipelines, Auto Loader, DABs.
• Proficiency in AWS cloud data engineering architectures.
• Strong SQL development and performance tuning.
• Experience with Git and CI/CD practices.

Responsibilities

• Refactor and migrate data models and ingestion pipelines to Databricks-based solutions.
• Migrate legacy data ingestion processes to modern patterns.
• Optimize Databricks cluster utilisation and implement cluster policies.
• Design, develop and maintain scalable data pipelines.
• Support QA, testing, and deployment activities within Agile delivery.

Skills

Databricks development

SQL development

Python programming

Cloud data engineering

Agile delivery

Tools

Databricks

AWS

Azure Data Factory (ADF)

Git

Spark

Delta Live Tables (DLT)

Databricks Asset Bundles (DABs)

Cluster Policy Management

Job description

Position Overview

The Data Developer Consultant will support the final phase of the organisation's Data Platform Modernisation Programme.

Position Details

Job Title: Data Developer (Databricks) Location: Abu Dhabi, UAE Work Model: Onsite (5 Days Per Week) Start Date: ASAP End Date: 31 October

Position Overview

The Data Developer Consultant will support the final phase of the organisation's Data Platform Modernisation Programme.

This role is focused on delivering the remaining technical activities required to:

• Complete the migration from legacy technologies.
• Standardise engineering practices.
• Prepare the platform for long-term operational support.
• Ensure full operational readiness before transition into business-as-usual (BAU) support.

Key Responsibilities

The successful consultant will be responsible for the following:

Data Platform Modernisation

• Refactor legacy data models and workflows to align with current engineering standards and best practices.
• Migrate data ingestion processes from legacy implementations to modern Databricks-based solutions.
• Support the decommissioning of legacy catalogues and platform components.
• Complete migration and remediation activities required to finalise the organisation's data platform transformation.

Databricks Engineering

• Review and optimise Databricks cluster utilisation.
• Implement cluster policies and performance optimisation improvements.
• Design, develop, and maintain scalable data pipelines.
• Develop and maintain data ingestion frameworks.
• Support the migration of ingestion processes to approved Databricks patterns and frameworks.

Development & Quality Assurance

• Participate in testing, validation, deployment activities, and code reviews.
• Ensure all deliverables meet defined requirements, standards, and acceptance criteria.
• Perform testing, validation, and quality assurance activities before delivery.
• Contribute to continuous improvement initiatives.

Documentation & Knowledge Transfer

• Document technical processes, standards, and operational workflows.
• Develop and maintain the required business context layer to support operational readiness.
• Contribute to knowledge transfer and support future operational teams.

Team Collaboration

• Support existing team members with:
  • New data ingestion pipelines
  • Data models
  • Platform enhancements
• Work closely with the Data Engineering team to ensure alignment with engineering standards and governance frameworks.
• Support activities required to transition the platform from implementation into steady-state operations.

Mandatory Experience

Required Skills & Experience

Databricks (Minimum 5 Years Required)

Candidates must have a minimum of 5 years' hands-on Databricks development experience, including the following:

• Delta Live Tables (DLT) / Spark Declarative Pipelines
• Auto Loader
• Databricks Asset Bundles (DABs)
• Databricks Cluster Optimization
• Cluster Policy Management

Cloud Environment

• Minimum 5 years of Databricks development experience on AWS environments.
• Strong understanding of cloud-based data engineering architectures.

Additional Technical Skills

Azure Data Factory (ADF)

• 1 to 3 years of experience.

Source Control

• Strong Git experience (mandatory).
• Strong understanding of source control best practices.

Database & Programming Skills

• Strong SQL development skills.
• Experience in SQL optimisation and performance tuning.
• Strong Python programming skills.

Testing & DevOps

• Experience with testing and validation practices.
• Experience with CI/CD deployment processes.
• Familiarity with DevOps methodologies and engineering practices.

The consultant will work within the existing data engineering organisation under the team's established Agile delivery framework.

Delivery Structure

• Scoped work packages will be assigned to the consultant.
• Each consultant will be assigned to and managed by an existing member of the Data Engineering team.
• Consultants will operate within established governance, standards, and delivery processes.
• Task prioritisation and workload management will be coordinated through the assigned team lead.

Agile Participation

Consultants Will Actively Participate In

• Sprint Planning
• Backlog Refinement
• Daily Stand-Ups
• Sprint Reviews
• Sprint Retrospectives
• Other Agile delivery ceremonies as required