Cloud Data Loading Architect (GCP and BigQuery)

Insight International (UK) Ltd; Leeds; Hybrid; GBP 60,000 - 80,000; Full time

Job summary

A leading cloud services firm is seeking a Cloud Data Loading Architect to design and implement automated data ingestion pipelines in Google Cloud Platform, focusing on BigQuery. The ideal candidate will have deep expertise in cloud data services, strong coding skills in SQL and Python, and a solid understanding of data governance. This role requires a blend of technical knowledge and strong communication skills to collaborate effectively with various teams and ensure efficient data processes. The position offers flexibility with a hybrid working model.

Qualifications

• Deep expertise in Google Cloud Data Services including BigQuery, Dataflow, and Pub/Sub.
• Hands-on experience designing data ingestion frameworks from various sources.
• Strong SQL and BigQuery optimization skills for efficient data processing.
• Proficient in Python or Java for data pipeline automation.
• Knowledge of cloud security and governance best practices.
• Experience with data quality frameworks and observability tools.
• Excellent communication skills for technical documentation and stakeholder engagement.

Responsibilities

• Design and implement ingestion pipelines for data loading into BigQuery.
• Define data loading frameworks and manage schema evolution.
• Create blueprints for governance and auditability of data.
• Establish data quality checks and validation rules.
• Optimize costs and access patterns in BigQuery.
• Collaborate with security teams on compliance and policies.
• Document technical processes and coach engineering teams.
• Troubleshoot data integration issues.

Skills

Google Cloud Data Services

Data Ingestion Engineering

SQL & BigQuery Optimization

Python

Cloud Security & Governance

Data Quality & Observability

Architectural Communication

Job description

Role: Cloud Data Loading Architect (GCP and BigQuery)

Location: Halifax or Leeds (Hybrid)

Job Type: Contract

Role Summary

• We are seeking an experienced Cloud Data Loading Architect to design, build, and optimise automated pipelines that ingest structured, semi‑structured, and unstructured datasets into Google Cloud Platform (GCP), specifically BigQuery.
• This role will lead end‑to‑end data ingestion design—from source discovery and schema mapping, through transformation and data quality, to scalable, secure loads into cloud‑native analytical warehouses.
• The ideal candidate combines strong cloud engineering skills with hands‑on data integration experience and a deep understanding of BigQuery performance optimisation.

Key Responsibilities

• Design and implement high‑throughput, fault‑tolerant ingestion pipelines for batch and streaming data landing in BigQuery, using Dataflow, Dataproc, Composer (Airflow), Pub/Sub, BigQuery Storage Write API, and related services.
• Define data loading frameworks, mapping rules, schema evolution strategy, and metadata management.
• Create reusable ingestion blueprints that ensure governance, lineage, and auditability.
• Establish data quality checks, validation rules, reconciliation logic, and SLAs.
• Optimize BigQuery cost, storage, partitioning, clustering, and access patterns.
• Collaborate with security & platform teams to ensure IAM, service accounts, VPCSC, and encryption policies are fully applied. Use GitHub, GitLab or Jenkins for CI/CD.
• Produce detailed technical documentation and coach engineering squads.
• Troubleshoot ingestion failures, performance bottlenecks, and cross‑platform data integration issues.

Qualifications

1. Deep Expertise in Google Cloud Data Services: BigQuery, GCS, Dataflow (Apache Beam), Pub/Sub, Dataproc, Cloud Composer, Storage Write API.
2. Data Ingestion Engineering Mastery: Hands‑on experience designing frameworks to load data from APIs, files, databases, event streams, and mainframe/legacy systems into cloud stores.
3. Strong SQL & BigQuery Optimisation skills: Partitioning, clustering, materialised views, cost‑efficient query design, columnar processing. Experience building transformation pipelines using Airflow, Dataflow, dbt, or equivalent orchestration tools. Ability to work with Parquet, Avro, ORC, JSON, CSV, nested/repeated structures, and schema evolution.
4. Strong Python and/or Java skills used to build Dataflow pipelines, ingestion utilities, automation scripts.
5. Cloud Security & Governance awareness: IAM roles, least‑privilege models, VPCSC, service accounts, artifact signing, audit. Cloud Build, GitHub Actions, Terraform, Cloud Deployment Manager or Pulumi.
6. Data Quality & Observability mindset: Experience implementing validation frameworks, anomaly detection, reconciliation rules, logging/monitoring (e.g., Cloud Logging, Cloud Monitoring).
7. Excellent Architectural Communication Skills: Ability to document, diagram, and communicate ingestion patterns to stakeholders at technical and non‑technical levels.