Senior Data Engineer

Intuition IT Solutions; Greater London; On-site; GBP 90,000 - 120,000; Full time

Job summary

Intuition IT Solutions seeks an experienced data platform engineer to own end‑to‑end AWS data pipelines. You will use Python/PySpark, SQL, and AWS services to build robust CDC and batch processes, ensuring quality, security, and governance across Lakehouse environments.

You will design scalable workflows with Step Functions/MWAA, implement IaC with Terraform, and advance CI/CD practices. The role emphasizes cost-aware optimization and collaboration with cross‑functional teams.

Qualifications

• 10+ years building data platforms on AWS.
• Strong Python (incl. PySpark) and SQL skills.
• Experience with data quality frameworks and CI/CD for data pipelines.

Responsibilities

• Ingest and CDC pipelines using AWS Lambda and EMR; manage schema evolution and backfills.
• Build and orchestrate workflows with Step Functions and MWAA, including retries and SLAs.
• Model datasets on S3 (Parquet/Iceberg) with Glue Data Catalog and Athena; optimize partitioning and performance.
• Apply data quality tests (Great Expectations/Deequ) and enforce data contracts.
• Implement security controls (IAM, KMS, VPC endpoints) and governance (Lake Formation) and audit trails.
• Drive observability: CloudWatch metrics/alerts, lineage, and cost-optimisation.
• Set up CI/CD and IaC (Terraform/CloudFormation/CDK); automate deployments with GitHub Actions.
• Create runbooks and maintain pipeline diagrams and SLAs/SLOs.
• Promote adoption and collaborate with other teams to reuse capabilities.

Skills

Python (PySpark)

SQL

AWS

Data engineering

Tools

DMS

Kinesis/MSK

Glue/EMR

Lambda

Step Functions/MWAA

Job description

• Ingestion and CDC: Implement pipelines using AWS Lambda , EMR for batch/streaming( future need); manage schema evolution and backfills.
• Orchestration: Build workflows with Step Functions and/or MWAA (Airflow), including retries, alerts, and SLAs.
• Lakehouse and warehouse: Model datasets on S3 (Parquet/Iceberg), Glue Data Catalog, Athena; partitioning, compaction, and performance tuning.
• Data quality and testing: Add automated tests (Great Expectations/Deequ, PySpark unit tests), data contracts, and CI gates.
• Security and governance: Implement IAM, KMS, VPC endpoints, Lake Formation policies, PII handling, audit trails (CloudTrail), and RBAC/RLS.
• Observability and FinOps: CloudWatch metrics/alerts, lineage, usage dashboards; cost optimisation (S3 lifecycle, compression, job sizing).
• CI/CD and IaC: Provision with Terraform/CloudFormation/CDK; build pipelines with GitHub Actions; environment promotion and rollbacks.
• Documentation and runbooks: Maintain pipeline diagrams, SLAs/SLOs, and incident playbooks.
• Adoption: be able to adopt capabilities from other teams instead of building duplicate capabilities and be able to contribute back to community.

Outcomes (first 60-90 days)

• Productionise one end to end pipeline with tests, monitoring, and alerting.
• Establish governance baseline (Lake Formation, tagging/classification, encryption) and data contracts for two key sources.
• Stand up CI/CD and IaC for data services; reduce at least one cost

Skills and experience

• 10+ years building data platforms on AWS; strong Python (incl. PySpark) and SQL.
• Hands on with DMS, Kinesis/MSK, Glue/EMR, Lambda, Step Functions/MWAA,
• IaC (Terraform) and CI/CD (GitHub Actions).
• Data quality frameworks (Great Expectations/Deequ), schema evolution, backfills.
• Security and compliance: IAM/KMS, private networking, Lake Formation,
• Performance and cost tuning across storage and compute.

Nice to have

• Snowflake, Apache Flink, Redshift Spectrum, OpenLineage, OPA/policy as