Cloud Data Engineer

United States Digital Space LLC; Greater London; Hybrid; GBP 70,000 - 110,000; Full time

Job summary

United States Digital Space LLC in London is seeking an experienced Cloud Data Engineer to design, build, and maintain scalable cloud-based data platforms and analytics solutions using AWS, Snowflake, dbt, PySpark, and SQL.

The role focuses on ETL/ELT development, data warehousing, performance optimization, and regulatory domain experience, with hybrid work blending company offices, client sites, and remote work as needed.

Qualifications

• Strong hands-on experience with AWS cloud platform.
• Experience in designing cloud-based data platforms.
• Strong ETL/ELT development experience.
• Experience building enterprise-grade data pipelines.

Responsibilities

• Design, develop, and maintain scalable data pipelines using AWS cloud services.
• Build and manage ETL/ELT workflows for data ingestion, transformation, validation, and reporting.
• Develop and maintain dbt models, tests, snapshots, macros, and documentation.
• Create high-performance data transformation solutions using PySpark and SQL.
• Design and optimize Snowflake data warehouse solutions.
• Integrate data from multiple sources including databases, APIs, cloud storage, and third-party systems.
• Implement data quality, reconciliation, audit controls, and exception handling.
• Optimize data processing jobs for performance, scalability, and cost efficiency.
• Develop CI/CD pipelines and automate deployment processes.
• Collaborate with architects, business analysts, QA teams, and stakeholders.
• Ensure data security, governance, compliance, and access controls.
• Support production deployment, monitoring, incident management, and troubleshooting.
• Prepare technical documentation and operational runbooks.

Skills

AWS

Snowflake

dbt

PySpark

SQL

ETL/ELT

Data Pipelines

RBAC security

CI/CD

Data Governance

Data Quality

Tools

Git

GitHub

Jenkins

AWS Glue

Snowpipe

CloudWatch

IAM

Job description

Cloud Data Engineer

About the Job you are considering:

We are seeking an experienced Cloud Data Engineer with expertise in AWS, Snowflake, dbt, PySpark, and SQL to design, build, and maintain scalable cloud-based data platforms and analytics solutions.

The ideal candidate should have strong experience in cloud data engineering, ETL/ELT development, data transformation, data warehousing, and performance optimization. Experience working in regulated industries such as Banking, Financial Services, Insurance, or Healthcare is preferred.

Hybrid working:

The places that you work from day to day will vary according to your role, your needs, and those of the business; it will be a blend of Company offices, client sites, and your home; noting that you will be unable to work at home 100% of the time.

Your Role:

• Design, develop, and maintain scalable data pipelines using AWS cloud services.
• Build and manage ETL/ELT workflows for data ingestion, transformation, validation, and reporting.
• Develop and maintain dbt models, tests, snapshots, macros, and documentation.
• Create high-performance data transformation solutions using PySpark and SQL.
• Design and optimize Snowflake data warehouse solutions.
• Integrate data from multiple sources including databases, APIs, cloud storage, and third‑party systems.
• Implement data quality, reconciliation, audit controls, and exception handling.
• Optimize data processing jobs for performance, scalability, and cost efficiency.
• Develop CI/CD pipelines and automate deployment processes.
• Collaborate with architects, business analysts, QA teams, and stakeholders.
• Ensure data security, governance, compliance, and access controls.
• Support production deployment, monitoring, incident management, and troubleshooting.
• Prepare technical documentation and operational runbooks.

Your Skills

Cloud & Data Engineering

• Strong hands‑on experience with AWS cloud platform.
• Experience in designing cloud‑based data platforms.
• Strong ETL/ELT development experience.
• Experience building enterprise‑grade data pipelines.

Snowflake

• Strong experience with Snowflake Data Warehouse.
• Experience in Snowflake performance tuning.
• Knowledge of Snowpipe, Streams, Tasks, and Stored Procedures.
• Experience implementing RBAC and security controls.

DBT

• Hands‑on experience with dbt Core or dbt Cloud.
• Development of reusable models, macros, tests, and snapshots.
• Data lineage, documentation, and dependency management.
• Integration with CI/CD pipelines.

PySpark & Programming

• Strong PySpark development expertise.
• Experience processing large‑scale datasets.
• Code optimization and performance tuning.
• Data transformation, enrichment, aggregations, and joins.

SQL

• Advanced SQL skills.
• Stored procedures, CTEs, window functions, and query optimization.
• Data warehousing and dimensional modelling concepts.

AWS Skills Expected

Candidate should have experience with one or more of the following AWS services:

• Amazon S3
• AWS Glue
• Amazon EMR
• AWS Lambda
• AWS Step Functions
• Amazon Redshift
• Amazon Athena
• Amazon RDS / Aurora
• AWS IAM
• AWS KMS
• Amazon CloudWatch
• AWS Secrets Manager

Data Modelling Skills

• Star Schema
• Snowflake Schema
• Dimensional Modeling
• Data Vault
• Data Warehouse Design
• Data Lake Architecture

Data Formats

Experience working with:

• Parquet
• Avro
• JSON
• CSV
• ORC

CI/CD & DevOps

• Git
• GitHub
• GitLab
• Bitbucket
• Jenkins
• GitHub Actions
• GitLab CI/CD
• AWS DevOps Tools

Domain Experience (Preferred)

Experience in one or more domains:

• Banking
• Financial Services
• Payments
• Cards
• Loans & Mortgages
• Risk & Compliance
• Regulatory Reporting
• AML
• Fraud Analytics
• Customer Data Platforms
• Financial Reporting
• Data Governance

We are a Disability Confident Employer

The company is proud to be a Disability Confident Employer (Level 2) under the UK Government’s Disability Confident scheme. As part of our commitment to inclusive recruitment, we will offer an interview to all candidates who declare a disability and meet the minimum essential criteria for the role.

Make It Real (what does it mean for you)

• You’ll be joining an accredited Great Place to work for Wellbeing in 2024. Employee wellbeing is vitally important to us as an organisation.
• To help support wellbeing we have trained "Mental Health Champions" across each of our business areas and invested in wellbeing apps such as Thrive and Peppy.
• You will be empowered to explore, innovate, and progress. You will benefit from the company’s learning for life mindset, meaning you will have countless training and development opportunities from thinktanks to hackathons, and access to 250,000 courses with numerous external certifications from AWS, Microsoft, Harvard ManageMentor, Cybersecurity qualifications and more.
• You will be joining one of the World’s Most Ethical Companies, as recognised by Ethisphere for 13 consecutive years.