Cloud Data Engineer — PySpark, Airflow & AWS

Athsai; United Kingdom; On-site; GBP 50,000 - 75,000; Full time

Job summary

Athsai is looking for an experienced and motivated Data Engineer in the United Kingdom. You will design and maintain scalable data platforms and pipelines that support analytics and machine learning initiatives.

The ideal candidate has strong expertise in Python, PySpark, and AWS services, and will collaborate with data analysts and software engineers to deliver high-performance data solutions. Knowledge of Terraform and CI/CD practices is also essential.

Qualifications

• Strong experience with Python and PySpark.
• Hands-on expertise with Apache Airflow.
• Extensive experience working with AWS cloud services.
• Experience with Terraform and Infrastructure as Code (IaC).
• Excellent troubleshooting, analytical, and problem-solving abilities.

Responsibilities

• Design, develop, and maintain scalable data pipelines using PySpark and Python.
• Build high-volume batch and real-time data processing solutions for large datasets.
• Develop, optimize, and monitor ETL/ELT workflows to ensure data quality.
• Implement workflow best practices to ensure reliability and maintainability.
• Optimize Spark applications for scalability and efficiency.

Skills

Python

PySpark

Apache Airflow

AWS services

Terraform

CI/CD pipelines

SQL skills

Distributed data processing

Tools

AWS Lambda

Amazon Redshift

AWS Glue

Amazon S3

Amazon EMR

GitHub Actions

Job description

Athsai is looking for an experienced and motivated Data Engineer in the United Kingdom. You will design and maintain scalable data platforms and pipelines that support analytics and machine learning initiatives.

The ideal candidate has strong expertise in Python, PySpark, and AWS services, and will collaborate with data analysts and software engineers to deliver high-performance data solutions. Knowledge of Terraform and CI/CD practices is also essential.