Senior Data Engineer: PySpark, Databricks & Lakehouse

Y-Axis; Dubai; On-site; AED 260,000 - 500,000; Full time

Job summary

Y-Axis in Dubai is seeking a Senior Data Engineer to design, develop and optimize scalable data pipelines for batch and real-time workloads. You will build ingestion frameworks and transform data using Python, PySpark, and Databricks within a Lakehouse architecture.

You will collaborate with data scientists, analysts and stakeholders to translate business needs into robust data solutions, ensuring quality, security and governance while optimizing Spark jobs and cloud resources.

Qualifications

• 7-10 years of experience in Data Engineering, Big Data, or Cloud Data Platforms.
• Strong programming experience in Python.
• Hands-on expertise in PySpark and Apache Spark.
• Extensive experience with Databricks, Delta Lake, and Lakehouse architecture.
• Strong proficiency in SQL and database performance optimization.
• Experience building enterprise-scale ETL/ELT pipelines.
• Strong hands-on experience with AWS Cloud Platform and cloud-native data engineering solutions.
• Experience with data modeling, schema design, and data warehousing concepts.
• Experience with workflow orchestration tools such as Apache Airflow or equivalent.
• Strong understanding of software engineering principles, version control, and testing methodologies.
• Experience using Git.

Responsibilities

• Design, develop, and maintain scalable batch and real-time data pipelines.
• Build data ingestion frameworks to process structured, semi-structured, and unstructured data from diverse sources.
• Develop and optimize data transformation workflows using Python, PySpark, and Databricks.
• Implement and manage ETL/ELT processes for large-scale analytics and reporting.
• Design and maintain Data Lakes, Lakehouse architectures, and Data Warehouses.
• Collaborate with Data Scientists, Business Analysts, Architects, and stakeholders to translate requirements into technical solutions.
• Ensure data quality, integrity, security, and governance across data platforms.
• Optimize Spark jobs, SQL queries, and cloud resources for performance and cost efficiency.
• Implement monitoring, logging, and alerting mechanisms for data pipelines.
• Contribute to architecture discussions, code reviews, and best practice implementation.

Skills

Python

PySpark

SQL

Spark

ETL/ELT pipelines

Data modeling

Data warehousing

Tools

Databricks

Delta Lake

Lakehouse architecture

AWS Cloud Platform

Apache Airflow

Git

Job description

Y-Axis in Dubai is seeking a Senior Data Engineer to design, develop and optimize scalable data pipelines for batch and real-time workloads. You will build ingestion frameworks and transform data using Python, PySpark, and Databricks within a Lakehouse architecture.

You will collaborate with data scientists, analysts and stakeholders to translate business needs into robust data solutions, ensuring quality, security and governance while optimizing Spark jobs and cloud resources.