AWS Glue Data Engineer — Lakehouse Ingestion

Venquis; Abu Dhabi; Vor Ort; AED 240.000 - 480.000; Vollzeit

Zusammenfassung

Venquis is seeking an AWS Glue Data Engineer to join the Data Factory Squad in Abu Dhabi/Dubai. You will build scalable ingestion pipelines, using AWS Glue and PySpark, to migrate source systems into the Lakehouse ingestion zone.

You will implement CDC, full/delta loads, and historical loading strategies, optimize performance, and automate deployments via Git and Terraform. Collaboration with Data Architect and Data Assurance is essential.

Qualifikationen

• Strong experience in AWS Glue, PySpark, and ETL pipeline development.
• Knowledge of Lakehouse architecture and Medallion design principles.
• Familiarity with CDC, delta loads, and historical data ingestion strategies.

Aufgaben

• Data Ingestion Development: Build and implement AWS Glue jobs for Bronze layer ingestion using defined standards and templates.
• Design and execute historical loading mechanisms to bring legacy data into the Lakehouse.
• Performance Optimization: Optimize Glue job performance (DPU allocation, parallelization, partitioning).
• Migration & Automation: Migrate source tables to Bronze layer with templates, later using AI-enabled acceleration; ensure version control and automated deployment via Git and Terraform.
• Governance & Monitoring: Ensure jobs comply with data contracts and are properly monitored; prepare documentation.

Kenntnisse

AWS Glue

PySpark

ETL pipeline development

Lakehouse architecture

Medallion design principles

CDC

delta loads

historical data ingestion

Tools

AWS Glue

S3

Athena

Lambda

Git

Terraform

Soda Core

Jobbeschreibung

Venquis is seeking an AWS Glue Data Engineer to join the Data Factory Squad in Abu Dhabi/Dubai. You will build scalable ingestion pipelines, using AWS Glue and PySpark, to migrate source systems into the Lakehouse ingestion zone.

You will implement CDC, full/delta loads, and historical loading strategies, optimize performance, and automate deployments via Git and Terraform. Collaboration with Data Architect and Data Assurance is essential.