Senior Data Engineer: Lakehouse & PySpark

Luxoft

Abu Dhabi

Presencial

AED 290.000 - 469.000

Jornada completa

14 días+

Una candidatura completa en un minuto

Currículum y carta de presentación personalizados, listos para enviar.

Supera los filtros ATS

Descripción de la vacante

Luxoft in Abu Dhabi is seeking an experienced Data Engineer to design, build, and operate data pipelines and models powering analytics across investment teams. You will own data end-to-end—from ingesting vendor and internal sources to modeling in the lakehouse and making data discoverable, reliable, and cost-efficient.

You will collaborate with BAs/PMs and quants, optimize pipelines using PySpark and Databricks, and help implement governance, testing, and production-ready datasets in complex

Formación

  • Bachelor’s degree in a related discipline.
  • 5+ years of data engineering experience.
  • Hands-on Databricks in production environments.
  • Strong PySpark and SQL in production.
  • Experience with Declarative Pipelines / orchestration on Databricks.
  • Solid understanding of ingestion, transformation, incremental processing, testing, and performance tuning.
  • Experience delivering production-ready datasets with monitoring and reliability patterns.
  • Familiarity with vendor market data feeds (e.g., Bloomberg, Refinitiv, MSCI, FactSet).

Responsabilidades

  • Participate in requirements clarification and sprint planning.
  • Design and implement ETL pipelines using PySpark.
  • Optimize ETL processes for performance and scalability.
  • Write unit and integration tests.
  • Support QA in acceptance process.
  • Resolve PROD incidents as a 3rd line engineer.

Conocimientos

PySpark
SQL
Databricks
Data pipelines
CI/CD
Data governance
Delta Lake
Data ingestion
ETL design

Educación

Bachelor’s degree in CS/Engineering/Information Systems

Herramientas

Palantir Foundry
Dynatrace
Datadog
Azure / AWS

Descripción del empleo

Project description

We’re looking for an experienced, hands-on Data Engineer who is capable of designing, building, and operating data pipelines and models that power analytics and applications across investment teams and Middle/Back office. The ideal candidate has financial markets familiarity (securities, prices, corporate actions, positions/holdings) and thrives in ambiguous environments—proactively shaping solutions, not waiting for tickets. You’ll own data end-to-end: from ingesting vendor and internal sources, to modeling in our lakehouse, to making data discoverable, reliable, and cost-efficient. You’ll partner closely with BAs/PMs and quants, anticipate downstream needs, and propose pragmatic architectures that balance speed, governance, and scalability.

Responsibilities

  • Participate in requirements clarification and sprint planning sessions.
  • Design technical solutions and implement them, inc ETL Pipelines – Build robust data pipelines in PySpark to extract, transform, using PySpark
  • Optimize ETL Processes – Enhance and tune existing ETL processes for better performance, scalability, and reliability
  • Writing unit and integration tests.
  • Support QA teammates in the acceptance process.
  • Resolving PROD incidents as a 3rd line engineer.

SKILLS

Must have

  • Bachelor’s degree (Computer Science, Engineering, Information Systems, or related discipline).
  • 5+ years experience in data engineering roles (flexible based on depth of capability).
  • Strong hands-on experience with Databricks in production environments (prerequisite).
  • Strong programming experience with PySpark (must) and strong SQL (must).
  • Proven experience with Declarative Pipelines / pipeline orchestration on Databricks (prerequisite).
  • Strong understanding of data engineering fundamentals: ingestion patterns, transformation design, incremental processing, testing, performance tuning.
  • Experience delivering production-ready datasets with appropriate operational controls (monitoring, troubleshooting, reliability patterns).
  • Experience with modern Lakehouse concepts (Delta tables, optimization strategies, file skipping, metadata/statistics awareness).
  • Exposure to data governance practices: cataloguing, documentation, business glossary/terms, lineage.
  • Experience working in enterprise environments with CI/CD pipelines and structured release processes.
  • Familiarity with vendor market data feeds (e.g., Bloomberg, Refinitiv, MSCI, FactSet) or similar multi-source mastering patterns.

Nice to have

• Strong Hands-on Expertise in Palantir Foundry. Proven experience with Foundry pipelines, ontologies, data lineage, transformations, and platform governance.• Proven Migration Experience from Palantir / to Databricks. Demonstrated experience leading or executing platform migrations, including pipeline conversion, data model redesign, and production cutover.• Familiarity with Dynatrace or Datadog for system observability and monitoring.• Databricks certification, cloud certifications (Azure/AWS), or enterprise data architecture certifications.

Consigue la evaluación confidencial y gratuita de tu currículum.

o arrastra y suelta tu archivo aquí

Similar jobs

Puestos de trabajo similares que vale la pena comparar

Senior Data Engineer (Investment Data)
Senior Data Engineer (Investment Data)

Luxoft Holding • Abu Dhabi

Presencial
AED 290.000 - 469.000
Senior Data Engineer (Investment Data)
Senior Data Engineer (Investment Data)

Luxoft • Abu Dhabi

Presencial
AED 391.000 - 614.000
Data Engineer
Data Engineer

VaporVM • Dubai

Presencial
AED 300.000 - 520.000
Data Engineer
Data Engineer

Amana Group • Abu Dhabi

Presencial
AED 210.000 - 330.000
Senior Data Engineer (Databricks, Lakehouse & AI-BI)
Senior Data Engineer (Databricks, Lakehouse & AI-BI)

NR Consulting • Abu Dhabi

Presencial
AED 300.000 - 520.000
Senior Data Engineer
Senior Data Engineer

Dicetek LLC • Dubai

Presencial
AED 360.000 - 600.000
Sr Data Platform Engineer
Sr Data Platform Engineer

Cactus International • Abu Dhabi

Híbrido
AED 360.000 - 540.000
Data Engineer
Data Engineer

Dicetek LLC • Abu Dhabi

Presencial
AED 240.000 - 380.000
Data Engineer
Data Engineer

Group AMANA • Dubai

Presencial
AED 280.000 - 420.000
Senior Technical Lead
Senior Technical Lead

HCL Technologies Limited • Dubai

Presencial
AED 167.000 - 279.000
Ubicaciones