Data Engineering Consultant

TN United Kingdom; London; On-site; GBP 50,000 - 90,000; Full time

Job summary

An established industry player is seeking a Data Engineering Consultant to design and optimize scalable data pipelines. This role is pivotal in transforming raw data into actionable insights while ensuring data governance and performance. You will architect real-time and batch data pipelines, utilize frameworks like Apache Spark and Snowflake, and collaborate with analysts and data scientists to deliver high-quality solutions. With a focus on automation and process optimization, you will mentor junior engineers and enforce best practices. Join a dynamic team that values diversity and offers a range of global benefits to support your career development and work-life balance.

Benefits

Competitive salary package

Company performance bonuses

Career coaching

Flexible working hours

Access to wellbeing apps

Training and certifications

Internal development programmes

Global career opportunities

Employee assistance programme

Inclusion and diversity programmes

Qualifications

• Expertise in programming languages like Python, SQL, and Java is essential.
• Experience with big data technologies and cloud platforms is crucial.

Responsibilities

• Develop and maintain data pipelines for efficient data handling.
• Collaborate with data teams to ensure alignment with business goals.

Skills

Python

SQL

Scala

Java

Apache Spark

Hadoop

Databricks

Snowflake

AWS

Azure

GCP

PostgreSQL

MongoDB

Docker

Kubernetes

Terraform

Job description

A Data Engineering Consultant designs, implements, and optimizes scalable data pipelines and architectures. This role bridges raw data and actionable insights, ensuring robustness, performance, and data governance. Collaboration with analysts and scientists is central to delivering high-quality solutions aligned with business objectives.

Key Responsibilities

1. Data Pipeline Development
   • Architect, implement and maintain real-time and batch data pipelines to handle large datasets efficiently.
   • Employ frameworks such as Apache Spark, Databricks, Snowflake or Airflow to automate ingestion, transformation, and delivery.
2. Data Integration & Transformation
   • Work with Data Analysts to understand source-to-target mappings and quality requirements.
   • Build ETL/ELT workflows, validation checks, and cleaning steps for data reliability.
3. Automation & Process Optimization
   • Automate data reconciliation, metadata management, and error-handling procedures.
   • Continuously refine pipeline performance, scalability, and cost-efficiency.
4. Collaboration & Leadership
   • Coordinate with Data Scientists, Data Architects, and Analysts to ensure alignment with business goals.
   • Mentor junior engineers and enforce best practices (version control, CI/CD for data pipelines).
   • Participate in technical presales activities and client engagement initiatives.
5. Governance & Compliance
   • Apply robust security measures (RBAC, encryption) and ensure regulatory compliance (GDPR).
   • Document data lineage and recommend improvements for data ownership and stewardship.

Qualifications

• Programming: Python, SQL, Scala, Java.
• Big Data: Apache Spark, Hadoop, Databricks, Snowflake, etc.
• Cloud: AWS (Glue, Redshift), Azure (Synapse, Data Factory, Fabric), GCP (BigQuery, Dataflow).
• Data Modelling & Storage: Relational (PostgreSQL, SQL Server), NoSQL (MongoDB, Cassandra), Dimensional modelling.
• DevOps & Automation: Docker, Kubernetes, Terraform, CI/CD pipelines for data flows.

Architectural Competencies

• Data Modelling: Designing dimensional, relational, and hierarchical data models.
• Scalability & Performance: Building fault-tolerant, highly available data architectures.
• Security & Compliance: Enforcing role-based access control (RBAC), encryption, and auditing.

Additional Information

Discover some of the global benefits that empower our people to become the best version of themselves:

• Finance: Competitive salary package, share plan, company performance bonuses, value-based recognition awards, referral bonus;
• Career Development: Career coaching, global career opportunities, non-linear career paths, internal development programmes for management and technical leadership;
• Learning Opportunities: Complex projects, rotations, internal tech communities, training, certifications, coaching, online learning platforms subscriptions, pass-it-on sessions, workshops, conferences;
• Work-Life Balance: Hybrid work and flexible working hours, employee assistance programme;
• Health: Global internal wellbeing programme, access to wellbeing apps;
• Community: Global internal tech communities, hobby clubs and interest groups, inclusion and diversity programmes, events and celebrations.

Our diversity makes us stronger - it drives meaningful change and enables us to build innovative technology solutions. We are committed to creating an inclusive community where all of us, regardless of background, identity, or personal characteristics, feels valued, respected, and free from discrimination. As an equal opportunity employer, we welcome applications from all individuals and base hiring decisions on merit, skills, qualifications, and potential.