Senior Data Platform Engineer (AWS, Lakehouse, CI/CD)

Intuition IT Solutions; Greater London; Sur place; GBP 90 000 - 120 000; Plein temps

Résumé du poste

Intuition IT Solutions seeks an experienced data platform engineer to own end‑to‑end AWS data pipelines. You will use Python/PySpark, SQL, and AWS services to build robust CDC and batch processes, ensuring quality, security, and governance across Lakehouse environments.

You will design scalable workflows with Step Functions/MWAA, implement IaC with Terraform, and advance CI/CD practices. The role emphasizes cost-aware optimization and collaboration with cross‑functional teams.

Qualifications

• 10+ years building data platforms on AWS.
• Strong Python (incl. PySpark) and SQL skills.
• Experience with data quality frameworks and CI/CD for data pipelines.

Responsabilités

• Ingest and CDC pipelines using AWS Lambda and EMR; manage schema evolution and backfills.
• Build and orchestrate workflows with Step Functions and MWAA, including retries and SLAs.
• Model datasets on S3 (Parquet/Iceberg) with Glue Data Catalog and Athena; optimize partitioning and performance.
• Apply data quality tests (Great Expectations/Deequ) and enforce data contracts.
• Implement security controls (IAM, KMS, VPC endpoints) and governance (Lake Formation) and audit trails.
• Drive observability: CloudWatch metrics/alerts, lineage, and cost-optimisation.
• Set up CI/CD and IaC (Terraform/CloudFormation/CDK); automate deployments with GitHub Actions.
• Create runbooks and maintain pipeline diagrams and SLAs/SLOs.
• Promote adoption and collaborate with other teams to reuse capabilities.

Connaissances

Python (PySpark)

SQL

AWS

Data engineering

Outils

DMS

Kinesis/MSK

Glue/EMR

Lambda

Step Functions/MWAA

Description du poste

Intuition IT Solutions seeks an experienced data platform engineer to own end‑to‑end AWS data pipelines. You will use Python/PySpark, SQL, and AWS services to build robust CDC and batch processes, ensuring quality, security, and governance across Lakehouse environments.

You will design scalable workflows with Step Functions/MWAA, implement IaC with Terraform, and advance CI/CD practices. The role emphasizes cost-aware optimization and collaboration with cross‑functional teams.