Senior Data Engineer – Python, PySpark, Cloud & Lakehouse | Abu Dhabi, UAE

Sightspectrum; United Arab Emirates; On-site; AED 300,000 - 540,000; Full time

Job summary

Sightspectrum is seeking a Senior Data Engineer in Abu Dhabi to join our enterprise Digitalization team. You will build and optimize data platforms, pipelines, and data products, working hands-on with Python, SQL, Spark/PySpark, ETL/ELT, and cloud/lakehouse technologies.

You will lead batch and streaming processing, implement CI/CD, ensure data governance and security, and mentor junior engineers while collaborating with stakeholders on data requirements and architecture decisions.

Qualifications

• Senior-level experience in Data Engineering with substantial enterprise-scale experience.
• Strong hands-on experience with Python, SQL, and Spark/PySpark.
• Proven experience developing enterprise data platforms and data pipelines.
• Strong knowledge of ETL/ELT processes and data modelling.
• Experience building data products and scalable data solutions.
• Strong understanding of cloud and lakehouse technologies.
• Experience with batch, streaming, and event-driven data processing.
• Strong knowledge of Git, CI/CD, automated testing, and observability.
• Experience with data platform performance optimization.
• Strong understanding of data security and governance.

Responsibilities

• Design and develop enterprise-scale data platforms and pipelines.
• Build robust data engineering solutions using Python, SQL, and Spark/PySpark.
• Develop and maintain ETL and ELT pipelines.
• Design scalable data models and enterprise data products.
• Implement cloud-based and lakehouse data architectures.
• Develop batch, streaming, and event-driven data processing solutions.
• Integrate data from multiple enterprise and operational sources.
• Implement automated testing and CI/CD practices for data engineering solutions.
• Use Git and modern software engineering practices for source control and deployment.
• Monitor data platforms using observability and performance-management practices.
• Troubleshoot, optimize, and improve data pipeline and platform performance.
• Implement appropriate data security and governance controls.
• Design scalable and reliable data processing architectures.
• Collaborate with business and technical stakeholders to understand data requirements.
• Provide technical leadership and guidance to data engineering teams.
• Mentor junior and mid-level data engineers.
• Support technical design, architecture decisions, and implementation standards.
• Contribute to enterprise digitalization and data modernization initiatives.

Skills

Python

SQL

Spark/PySpark

ETL/ELT

Data modelling

Cloud computing

Lakehouse architectures

Batch processing

Streaming data

Event-driven architectures

CI/CD

Git

Observability

Automated testing

Data governance

Data security

Tools

Databricks

Microsoft Azure

AWS

Snowflake

Apache Kafka

Apache Airflow

Job description

Location

Abu Dhabi, United Arab Emirates

Job Category

• Information Technology (IT) & Software
• Engineering & Technical
• Science & Research
• Energy, Oil & Gas

Job Overview

We are seeking an experienced Senior Data Engineer to join an enterprise Digitalization team in Abu Dhabi. The role is designed for a highly hands-on data engineering professional with substantial experience building and optimizing enterprise-scale data platforms, data pipelines, and data products.

The successful candidate will work across Python, SQL, Spark/PySpark, ETL/ELT, data modelling, cloud and lakehouse technologies, batch and streaming data processing, and event-driven architectures. The position also requires strong capabilities in performance optimization, security, data governance, automated testing, observability, and technical design.

This opportunity offers strong potential for career growth and professional development through enterprise data engineering, cloud platforms, modern lakehouse architectures, and digitalization initiatives. Experience with industrial data, IoT, SCADA, telemetry, time-series data, digital twins, or predictive analytics can provide valuable domain exposure. Relevant cloud, data engineering, or platform certifications and technical training can further support career development.

Key Responsibilities

• Design and develop enterprise-scale data platforms and pipelines.
• Build robust data engineering solutions using Python, SQL, and Spark/PySpark.
• Develop and maintain ETL and ELT pipelines.
• Design scalable data models and enterprise data products.
• Implement cloud-based and lakehouse data architectures.
• Develop batch, streaming, and event-driven data processing solutions.
• Integrate data from multiple enterprise and operational sources.
• Implement automated testing and CI/CD practices for data engineering solutions.
• Use Git and modern software engineering practices for source control and deployment.
• Monitor data platforms using observability and performance-management practices.
• Troubleshoot, optimize, and improve data pipeline and platform performance.
• Implement appropriate data security and governance controls.
• Design scalable and reliable data processing architectures.
• Collaborate with business and technical stakeholders to understand data requirements.
• Provide technical leadership and guidance to data engineering teams.
• Mentor junior and mid-level data engineers.
• Support technical design, architecture decisions, and implementation standards.
• Contribute to enterprise digitalization and data modernization initiatives.

Requirements & Qualifications

• Senior-level experience in Data Engineering with substantial enterprise-scale experience.
• Strong hands-on experience with Python, SQL, and Spark/PySpark.
• Proven experience developing enterprise data platforms and data pipelines.
• Strong knowledge of ETL/ELT processes and data modelling.
• Experience building data products and scalable data solutions.
• Strong understanding of cloud and lakehouse technologies.
• Experience with batch, streaming, and event-driven data processing.
• Strong knowledge of Git, CI/CD, automated testing, and observability.
• Experience with data platform performance optimization.
• Strong understanding of data security and governance.
• Experience with one or more of the following is highly desirable:
  • Databricks
  • Microsoft Azure
  • AWS
  • Snowflake
  • Apache Kafka
  • Apache Airflow
• Experience mentoring or technically leading data engineers.
• Strong technical design and stakeholder-management capabilities.
• Ability to design, code, test, troubleshoot, and optimize data engineering solutions independently.

Preferred Domain Experience

Experience with any of the following will be a strong advantage:

• IoT data
• SCADA systems
• Historian data
• Telemetry data
• Time-series data
• Asset data
• Digital twins
• Predictive analytics
• Industrial environments
• Energy-sector environments

Salary, Benefits & Career Growth

Salary details have not been provided for this position.

The role provides opportunities for career progression and professional development through enterprise data engineering, cloud and lakehouse technologies, digitalization, industrial data platforms, and technical leadership. Professionals can further strengthen their expertise through hands-on experience, specialized data engineering and cloud training, professional certifications, and continuous technical upskilling.