Senior Data Engineer: Lakehouse & PySpark

Luxoft; Abu Dhabi; On-site; AED 290,000 - 469,000; Full time

Job summary

Luxoft in Abu Dhabi is seeking an experienced Data Engineer to design, build, and operate data pipelines and models powering analytics across investment teams. You will own data end-to-end—from ingesting vendor and internal sources to modeling in the lakehouse and making data discoverable, reliable, and cost-efficient.

You will collaborate with BAs/PMs and quants, optimize pipelines using PySpark and Databricks, and help implement governance, testing, and production-ready datasets in complex

Qualifications

• Bachelor’s degree in a related discipline.
• 5+ years of data engineering experience.
• Hands-on Databricks in production environments.
• Strong PySpark and SQL in production.
• Experience with Declarative Pipelines / orchestration on Databricks.
• Solid understanding of ingestion, transformation, incremental processing, testing, and performance tuning.
• Experience delivering production-ready datasets with monitoring and reliability patterns.
• Familiarity with vendor market data feeds (e.g., Bloomberg, Refinitiv, MSCI, FactSet).

Responsibilities

• Participate in requirements clarification and sprint planning.
• Design and implement ETL pipelines using PySpark.
• Optimize ETL processes for performance and scalability.
• Write unit and integration tests.
• Support QA in acceptance process.
• Resolve PROD incidents as a 3rd line engineer.

Skills

PySpark

SQL

Databricks

Data pipelines

CI/CD

Data governance

Delta Lake

Data ingestion

ETL design

Education

Bachelor’s degree in CS/Engineering/Information Systems

Tools

Palantir Foundry

Dynatrace

Datadog

Azure / AWS

Job description

Luxoft in Abu Dhabi is seeking an experienced Data Engineer to design, build, and operate data pipelines and models powering analytics across investment teams. You will own data end-to-end—from ingesting vendor and internal sources to modeling in the lakehouse and making data discoverable, reliable, and cost-efficient.

You will collaborate with BAs/PMs and quants, optimize pipelines using PySpark and Databricks, and help implement governance, testing, and production-ready datasets in complex