Senior Data Engineer

FPSG; Glasgow; On-site; GBP 75,000 - 110,000; Full time

Job summary

FPSG in Glasgow seeks a Senior Data Engineer to own the Databricks-based data platform's maintenance, expansion, and performance. This hands-on role centers on writing pipeline code, building jobs, and data modelling as the daily core tasks.

You design and operate pipelines, collaborate with source systems, and implement Delta Lake pipelines while controlling deployment bundles and CI practices.

Qualifications

• Strong SQL and data modelling skills.
• Hands-on experience with Azure data platforms in production.
• Experience building Databricks Pipelines and Jobs in production.
• Background with Snowflake and interest in Databricks.
• Ability to design silver/gold tables and medallion patterns.

Responsibilities

• Designing, building, and operating Databricks Pipelines and Jobs.
• Designing and building Delta Lake pipelines.
• Owning Databricks deployment bundles, packaging, versioning, and promoting code.
• Building environment-variable-driven Azure Data Factory pipelines.
• Building the data layer that feeds Microsoft Fabric/OneLake.
• Monitoring and troubleshooting data pipelines for performance, reliability and cost.
• Taking ownership of data solutions from source systems through to business outcomes.

Skills

SQL

Python

Scala

Data modelling

Ownership mindset

Tools

Databricks Pipelines & Jobs

Delta Lake

Azure Data Factory

Databricks Asset Bundles

Snowflake

Job description

Exciting opportunity for a Senior Data Engineer to own the maintenance, expansion, and performance of Databricks-based data platform. This is a hands-on role, you'll be writing pipeline code, building jobs, and modelling as the main part of your day to day.

The Role

You will be the technical expert for the data solution, to work alongside the engineering lead to manage the data aspects of the digital solution. You will be designing and building data pipelines, working with source systems, developing Databricks jobs, manipulating and modelling data, and making technical decisions about how data should be managed and structured. You’ll work closely with the wider engineering team to understand what the business needs from its data, what the end goal is, and how best to get there.

Key responsibilities will include:

• Designing, building, and operating Databricks Pipelines and Jobs
• Designing and building Delta Lake pipelines
• Owning Databricks deployment bundles, packaging, versioning, and promoting pipeline code and job definitions
• Building and maintaining environment-variable-driven Azure Data Factory pipelines
• Building the data layer that feeds Microsoft Fabric/OneLake
• Monitoring and troubleshooting data pipelines for performance, reliability and cost.
• Taking ownership of data solutions from initial source-system investigation through to the final business outcome.

About You

We’re looking for someone who can demonstrate real-world, hands-on data engineering experience.

You’ll ideally have:

• Strong, demonstrable experience working with Azure data platforms in production.
• Hands-on experience building and running Databricks Pipelines and Jobs in a production environment. Candidates with a strong background in Snowflake and drive to learn Databricks also encouraged to apply.
• Strong data modelling and data manipulation skills, with the ability to take raw source data and turn it into sensible, structured datasets.
• Strong data modelling ability, able to look at raw source data and design sensible, well-structured silver/gold tables
• Strong understanding of medallion (bronze/silver/gold) design patterns, and data architecture right at table level
• Solid Delta Lake knowledge, deletion vectors, time travel, optimization, compatibility considerations
• Experience with Databricks Asset Bundles or equivalent deployment/CI practices for data pipelines
• Practical experience building environment-variable/parameter-driven Azure Data Factory pipelines
• Proven experience of managing large data architectures
• Strong SQL, and comfort in at least one of Python/Scala for pipeline logic

If you’re a Senior Data Engineer who enjoys getting into the detail, solving complex data problems and having genuine ownership of the solution, we’d be keen to hear from you.

FPSG is committed to equal opportunities regardless of gender, race, disability, sexual orientation, religion or belief and age.