Google Cloud Platform Data Engineer

Astro Studios, Inc.; Bristol; On-site; GBP 90,000 - 130,000; Full time

Benefits offered by this job

Private medical insurance

Season ticket loan

25 days annual leave

Pension scheme

Annual bonus

Life and income protection

Tax benefits

Voluntary benefits

Job summary

PA Consulting is hiring a Principal GCP Data Engineer in Bristol to lead data engineering efforts for complex client programmes. You will own end-to-end data solutions, guide architecture, and drive best practices across teams with a focus on production-grade pipelines and cloud-native patterns.

You will mentor teammates, influence client stakeholders, and contribute to design discussions beyond data processing, shaping the overall data strategy for large-scale engagements.

Qualifications

• Experience delivering and deploying production-ready data processing solutions using BigQuery, Pub/Sub, Dataflow and Dataproc.
• End-to-end solutions with batch and streaming via Spark and Beam.
• Expert understanding of relational and non-relational data stores and data warehousing/data lakes.
• Data pipeline patterns: ETL/ELT, event-driven architectures, stream processing.
• Translate business requirements into technical specifications and designs.
• Experience with metadata management products (Data Catalog, Collibra) and Data Governance.
• Developing solutions on GCP with cloud-native principles and patterns.
• Data quality alerts and quarantine solutions for trusted downstream datasets.
• CI/CD pipelines with git, automated tests and deployments.
• Agile teams using Scrum or Kanban.

Responsibilities

• Develop robust data processing jobs using Google Cloud tools (Dataflow, Dataproc, BigQuery).
• Design and deliver automated data pipelines with Cloud Composer.
• Contribute to end-to-end architecture discussions beyond data processing.
• Own development process for the team, with robust patterns for architecture, scope, quality and deployments.
• Define and evolve PA’s data engineering standards and practices.
• Lead technical discussions with client stakeholders to gain buy-in.
• Coach and mentor team members across seniority levels.

Skills

BigQuery

Pub/Sub

Dataflow

Dataproc

Apache Spark

Apache Beam

Data warehousing

Data lakes

Event-driven architectures

ETL/ELT

Cloud Data Catalog

Collibra

GCP

CI/CD

Git

Agile (Scrum/Kanban)

Kafka

PostgreSQL

MySQL

Oracle

SQL Server

Dataplex

Tools

Cloud Composer

Kafka

PostgreSQL

MySQL

Oracle

SQL Server

Dataplex

Job description

As a Principal GCP Data Engineer, you'll be a true subject matter expert in using the data processing and management capabilities of Google Cloud to develop data-driven solutions for our clients.

You will typically lead a team or the solution delivery effort, demonstrating technical excellence through leading by example. You could be providing technical support, leading an engineering team or working across multiple teams as a subject matter expert who is critical to the success of a large programme of work.

Your team members will look to you as a trusted expert and will expect you to define the end-to-end software development lifecycle in line with modern best practices. As part of your responsibilities, you will be expected to:

• Develop robust data processing jobs using tools such as Google Cloud Dataflow, Dataproc and BigQuery
• Design and deliver automated data pipelines that use orchestration tools such as Cloud Composer
• Design end-to-end solutions and contribute to architecture discussions beyond data processing
• Own the development process for your team, building strong principles and putting robust methods and patterns in place across architecture, scope, code quality and deployments.
• Shape team behaviour for writing specifications and acceptance criteria, estimating stories, sprint planning and documentation.
• Actively define and evolve PA’s data engineering standards and practices, ensuring we maintain a shared, modern and robust approach.
• Lead and influence technical discussions with client stakeholders to achieve the collective buy‑in required to be successful
• Coach and mentor team members, regardless of seniority, and work with them to build their expertise and understanding.

Qualifications

To be successful in this role, you will need to have:

• Experience delivering and deploying production‑ready data processing solutions using BigQuery, Pub/Sub, Dataflow and Dataproc
• Experience developing end‑to‑end solutions using batch and streaming frameworks such as Apache Spark and Apache Beam.
• Expert understanding of when to use a range of data storage technologies including relational/non‑relational, document, row‑based/columnar data stores, data warehousing and data lakes.
• Expert understanding of data pipeline patterns and approaches such as event‑driven architectures, ETL/ELT, stream processing and data visualisation.
• Experience working with business owners to translate business requirements into technical specifications and solution designs that satisfies the data requirements of the business.
• Experience working with metadata management products such as Cloud Data Catalog and Collibra and Data Governance tools like Dataplex
• Experience in developing solutions on GCP using cloud‑native principles and patterns.
• Experience building data quality alerting and data quarantine solutions to ensure downstream datasets can be trusted.
• Experience implementing CI/CD pipelines using techniques including as git code control/branching, automated tests and automated deployments.
• Comfortable working in an Agile team using Scrum or Kanban methodologies.

In addition to the above, we would be thrilled if you also had:

• Experience of working on migrations of enterprise scale data platforms including Hadoop and traditional data warehouses
• An understanding of machine learning model development lifecycle, feature engineering, training and testing
• Good understanding or hands‑on experience of Kafka
• Experience as a DBA or developer on RDBMS such as PostgreSQL, MySQL, Oracle or SQL Server
• Experience designing data applications to meet non‑functional requirements such as performance and availability
• You are pragmatic and already understand that writing code is only part of what a data engineer does.
• You can clearly communicate with both clients and peers, describing technical issues and solutions in both written and meeting/workshop contexts.
• You are able to clearly explain technical concepts to non technical audiences at all levels of an organisation.
• You are able to influence and persuade senior and specialist client stakeholders, potentially across multiple organisational boundaries without direct authority.
• You are a confident problem solver and troubleshooter.
• You are confident and generous in sharing your specialist knowledge, ideas and solutions.
• You are constantly learning and able to make others better by consciously teaching and unconsciously inspiring.

Additional information

Benefits package at PA:

• Private medical insurance
• Interest free season ticket loan
• 25 days annual leave with the opportunity to buy 5 additional days
• Company pension scheme
• Annual performance‑based bonus
• Life and Income protection insurance
• Tax efficient benefits (cycle to work, give as you earn, childcare benefits)
• Voluntary benefits (Dental, critical illness, spouse/partner life assurance)