Senior Data Engineer

Nova Hunte Ltd; Greater London; Remote; GBP 90,000 - 130,000; Full time

Benefits offered by this job

Remote work

Long-term contract

Job summary

Nova Hunte Ltd is seeking a Senior Data Engineer to design, build and maintain scalable data pipelines using Python and PySpark in a remote, long-term contract with a London-based client.

You will work on AWS data services, data lakes, ETL/ELT, data modelling, and real-time processing, collaborate with product teams, mentor juniors, and ensure performance, security and observability across the data platform.

Qualifications

• 8+ years of data engineering experience.
• Proficient in Python and PySpark; strong SQL and data modelling.
• Extensive AWS data services experience and data lake/platform design.
• Experience with NoSQL databases, OpenSearch/Elasticsearch, and streaming tools.
• Strong ETL/ELT, data governance, and performance optimization.

Responsibilities

• Design and develop large-scale data pipelines with Python and PySpark.
• Build batch and near real-time processing on AWS data platforms.
• Design scalable data architectures and data lake solutions.
• Implement ETL/ELT workflows for ingesting, transforming, and publishing datasets.
• Collaborate with cross-functional teams and mentor junior engineers.

Skills

Python

PySpark

SQL

Data modelling

Analytics

Mentoring

Education

Bachelor's or Master's in CS/Engineering

Tools

AWS (EMR, Glue, Lambda, S3, Athena, Redshift)

OpenSearch/Elasticsearch

DynamoDB

Docker

Kubernetes

Airflow

Kafka/Kinesis

Terraform

Databricks

Job description

# Senior Data EngineerIndia ContractAre you a Senior Data Engineer with 8+ years of experience in building scalable data platforms and pipelines? Apply now for a remote long-term contract opportunity with a leading client in London.## Requirements* 8+ years of experience in Data Engineering* Strong expertise in Python and extensive experience with PySpark and distributed data processing* Expert-level knowledge of AWS services including EMR, Glue, Lambda, S3, Athena, Redshift, ECS/EKS, Step Functions, EventBridge, IAM, CloudWatch, and SNS/SQS* Hands-on experience building data lakes and modern data platforms* Strong understanding of ETL/ELT frameworks and data modelling* Strong SQL skills and query optimization experience* Experience with NoSQL databases, particularly DynamoDB, including partitioning, indexing, and data access patterns* Hands-on experience with OpenSearch/Elasticsearch, including designing indexes, search schemas, query optimisation, and search performance tuning* Working knowledge of Java and understanding of object-oriented programming and software engineering best practices* Experience implementing analytics and monitoring dashboards* Strong problem-solving and analytical skills* Ability to work in a fast-paced, cloud-native environment* Excellent communication and stakeholder management skills* Strong ownership and accountability mindset with ability to mentor and guide team members* (Desirable) Experience with streaming platforms such as Kafka or Kinesis* (Desirable) Experience with Airflow or other workflow orchestration tools* (Desirable) Experience with Docker and Kubernetes* (Desirable) Exposure to Data Lake and Lakehouse architecture* (Desirable) Experience in Agile/Scrum development environments* (Desirable) Experience with Apache Kafka, Apache Airflow, Terraform, Databricks, Delta Lake, Snowflake, CI/CD pipelines, or Machine Learning data pipelines* (Desirable) Bachelor's or Master's degree in Computer Science, Engineering, or a related field* Language(s): English (fluent)## Job Description* Design, develop, and maintain large-scale data pipelines using Python and PySpark* Build and optimise batch and near real-time data processing solutions* Design scalable data architectures on AWS* Develop ETL/ELT workflows for ingesting, transforming, and publishing datasets* Work with structured and semi-structured data from multiple sources* Design and maintain NoSQL database solutions using DynamoDB* Implement search and analytics solutions using OpenSearch* Optimise data processing jobs for performance, scalability, and cost efficiency* Collaborate with product teams, architects, and software engineers to define data requirements* Monitor, troubleshoot, and resolve data platform and pipeline issues* Implement data quality, governance, security, and observability best practices* Participate in code reviews and mentor junior engineers