Lead Data Engineer

Harvey Nash Group; City of Edinburgh; Hybrid; GBP 92,000 - 129,000; Full time

Job summary

Harvey Nash Group is seeking an experienced Lead Data Engineer for a 6-month contract (Inside IR35) with hybrid working in Edinburgh/Glasgow. You will design, develop, and support scalable AWS data warehouse and data lake solutions, delivering end-to-end data pipelines and analytics platforms in an Agile environment.

You will implement serverless architectures, ETL/ELT pipelines, and dashboards, collaborating with stakeholders to deliver BI-ready data assets and governance practices.

Qualifications

• Experience with AWS data and analytics services such as Redshift, Glue, S3, Athena, Lambda, Step Functions, RDS/Aurora, DynamoDB, DMS, EMR, CloudWatch and IAM.
• Strong programming experience in Python and SQL.
• Experience with Spark (PySpark preferred) and large-scale data processing.
• Experience developing ETL/ELT pipelines and data warehouse solutions.
• Knowledge of API integration and microservices architectures.
• Experience working in Agile/Scrum delivery environments.

Responsibilities

• Design, build, and maintain cloud-based data warehouse and data lake solutions on AWS.
• Develop and support ETL/ELT processes for data ingestion, transformation, and integration.
• Build and optimise data pipelines using Python, SQL, Spark, and AWS services.
• Develop serverless and event-driven solutions using AWS Lambda, Step Functions, EventBridge, SQS, and SNS.
• Support and enhance existing data platforms, applications, and integrations.
• Work closely with business and technical stakeholders to deliver solutions in 2-week Agile sprints.
• Create and maintain dashboards, reporting, and monitoring solutions using QuickSight, Power BI, Grafana, and ELK.
• Implement data quality, security, governance, and operational monitoring best practices.

Skills

Python

SQL

Spark

ETL/ELT

Agile/Scrum

Data warehousing

Data governance

Cloud analytics

AWS

Tools

AWS

Redshift

S3

Athena

Lambda

Step Functions

RDS/Aurora

DynamoDB

DMS

EMR

CloudWatch

IAM

Job description

Lead Data Engineer | 6 Month Contract | (Inside IR35) | Hybrid, 2 days pw Edinburgh/Glasgow | Starting ASAP

Day Rate: £DOE

About the role:

Harvey Nash's Pub Sec client is seeking an experienced AWS Data Engineer to design, develop, and support scalable data warehouse and cloud-based data solutions. The successful candidate will work within an Agile Scrum team, delivering end-to-end data engineering solutions, ETL pipelines, and analytics platforms that support business intelligence and digital applications.

You will:

• Design, build, and maintain cloud-based data warehouse and data lake solutions on AWS.
• Develop and support ETL/ELT processes for data ingestion, transformation, and integration.
• Build and optimise data pipelines using Python, SQL, Spark, and AWS services.
• Develop serverless and event-driven solutions using AWS Lambda, Step Functions, EventBridge, SQS, and SNS.
• Support and enhance existing data platforms, applications, and integrations.
• Work closely with business and technical stakeholders to deliver solutions in 2-week Agile sprints.
• Create and maintain dashboards, reporting, and monitoring solutions using QuickSight, Power BI, Grafana, and ELK.
• Implement data quality, security, governance, and operational monitoring best practices.

Essential Criteria

Strong experience with AWS data and analytics services including:

• Redshift
• Glue
• S3
• Athena
• Lambda
• Step Functions
• RDS/Aurora
• DynamoDB
• DMS
• EMR
• CloudWatch
• IAM

Strong programming experience in Python and SQL.

Experience with Spark (PySpark preferred) and large-scale data processing.

Experience developing ETL/ELT pipelines and data warehouse solutions.

Knowledge of API integration and microservices architectures.

Experience working in Agile/Scrum delivery environments.

This role has been deemed Inside IR35 by the client. Applicants must hold, or be happy to apply for, a valid Basic Disclosure Scotland.