Senior Data Engineer: Production Pipelines on Fabric & PySpark

F5 Remote Jobs; City of Edinburgh; Hybrid; GBP 65,000 - 105,000; Full time

Benefits offered by this job

Flexible working

Hybrid/Remote options

Edinburgh office hub

Job summary

CreateFuture is seeking a Senior Data Engineer to lead production data pipelines built on Microsoft Fabric, Python, and PySpark. You will translate data requirements into robust, maintainable solutions while mentoring engineers and shaping architecture with cross-functional teams.

You’ll own CI/CD pipelines, troubleshoot complex issues, and collaborate with clients to deliver scalable data platforms across engagements. This role emphasizes autonomy and technical leadership.

Qualifications

• Deep SQL skills and data modelling experience.
• Strong Python and PySpark for batch ELT processing.
• Experience designing and operating production data pipelines.

Responsibilities

• Design, build, and maintain scalable, production-ready data pipelines.
• Decide on pipeline architecture and data modelling trade-offs.
• Mentor junior engineers and review code to raise delivery standards.
• Own CI/CD pipelines for data workflows and deployments.
• Troubleshoot complex live data issues.
• Collaborate with cross-functional teams and clients to shape data needs.

Skills

SQL

Python

PySpark

CI/CD

Mentoring

Tools

Microsoft Fabric

Databricks

Snowflake

Synapse

Job description

CreateFuture is seeking a Senior Data Engineer to lead production data pipelines built on Microsoft Fabric, Python, and PySpark. You will translate data requirements into robust, maintainable solutions while mentoring engineers and shaping architecture with cross-functional teams.

You’ll own CI/CD pipelines, troubleshoot complex issues, and collaborate with clients to deliver scalable data platforms across engagements. This role emphasizes autonomy and technical leadership.