Senior Data Engineer - Contract

TN United Kingdom; Brynmawr; On-site; GBP 50,000 - 90,000; Full time

Job summary

Join a forward-thinking company as a Senior Data Engineer, where your expertise will drive the development of sophisticated data infrastructure across hybrid environments. You will design and manage data pipelines, ensuring seamless integration and accessibility of data for advanced analytics. This role emphasizes ethical data use and innovative solutions, fostering a collaborative environment. If you are passionate about data and technology, this is your chance to make a significant impact while working alongside dedicated professionals in a supportive atmosphere.

Qualifications

• 5+ years of experience in data engineering with a focus on hybrid environments.
• Strong Python skills for scripting and automation of data processes.

Responsibilities

• Design and maintain data pipelines for structured and unstructured data.
• Implement data storage solutions using Azure and on-premises infrastructure.

Skills

Python

ETL/ELT

Hybrid Cloud Data Architecture

Containerisation and Orchestration

Workflow Automation

Event Streaming

Data Security

Search and Database Management

Education

Bachelor's Degree in Computer Science or related field

Certifications in Azure

Tools

Docker

Kubernetes

Apache Airflow

Kafka

Keycloak

Elasticsearch

PostgreSQL

Job description

Social network you want to login/join with:

col-narrow-left

Client:

Methods

Location:

Brynmawr, United Kingdom

Job Category:

-

EU work permit required:

Yes

col-narrow-right

Job Reference:

f779f006f7b1

Job Views:

10

Posted:

26.04.2025

Expiry Date:

10.06.2025

col-wide

Job Description:

Methods Analytics

Methods Analytics (MA) exists to improve society by helping people make better decisions with data. Combining passionate people, sector-specific insight, and technical excellence to provide our customers an end-to-end data service.

We use a collaborative, creative, and user-centric approach to data to do good and solve difficult problems. Ensuring that our outputs are transparent, robust, and fostering discussion and debate as part of our methodology. We will question assumptions, ambition, and process – but do so with respect and humility.

We relish difficult problems and overcome them with innovation, creativity, and technical freedom to design optimal solutions. Ethics, privacy, and quality are at the core of our work, and we will not compromise these for outcomes.

We treat data with respect and use it only for the right purpose. Our team is positive, dedicated, and relentless. Data is a vast topic, but we aim for interactions that are engaging, informative, and fun in equal measure, while maintaining a focus on outcomes and delivering quality products for our customers.

We are passionate about our people; we want our colleagues to develop their strengths and enjoy their work.

Methods was acquired by the Alten Group in early 2022.

Requirements

On-site, Full-time.

We are seeking a seasoned Senior Data Engineer (Infrastructure) to join our team. This role is essential for designing, building, and maintaining sophisticated data infrastructure systems across on-premises and Azure cloud environments. The position involves deploying and managing scalable data operations that support advanced analytics and data-driven decision-making, crucial for our organizational growth and innovation.

Responsibilities

• Develop and Manage Data Pipelines: Design, construct, and maintain efficient and reliable data pipelines using Python, supporting streaming and batch data processing across structured, semi-structured, and unstructured data in on-premises and Azure environments.
• Hybrid Cloud and Data Storage Solutions: Implement and manage data storage solutions leveraging both on-premises infrastructure and Azure, ensuring seamless data integration and accessibility.
• Containerisation and Orchestration: Use Docker for containerisation and Kubernetes for orchestration, ensuring scalable deployment across various environments.
• Workflow Automation: Employ tools like Apache Airflow to automate data workflows within hybrid environments.
• Event Streaming Experience: Utilize event-driven technologies such as Kafka for real-time data streams.
• Security and Compliance: Manage security setups and access controls, incorporating tools like Keycloak to protect data and ensure legal compliance.
• Data Search and Analytics: Oversee Elasticsearch setups for data searching and analytics in mixed infrastructures.
• Database Management: Administer and optimize PostgreSQL databases for high performance and availability.

Essential Skills and Experience

• Strong Python Skills: Expertise in Python scripting and automation for data processes.
• Experience with ETL/ELT: Proven experience developing and optimizing workflows, especially in hybrid environments.
• Hybrid Cloud Data Architecture: Deep knowledge of integrating on-premises infrastructure with Azure cloud services.
• Containerisation and Orchestration: Experience with Docker and Kubernetes managing applications across environments.
• Workflow Automation: Practical experience with Apache Airflow in hybrid data settings.
• Event Streaming: Ability to manage platforms like Kafka.
• Data Security: Knowledge of security practices and tools like Keycloak.
• Search and Database Management: Skills in Elasticsearch and PostgreSQL in hybrid environments.

Your Impact

In this role, you will enable business leaders to make informed decisions by delivering accurate, timely data insights from a robust hybrid infrastructure. Your expertise will enhance the integration and scalability of data operations, promoting a data-driven culture and mentoring team members.

Desirable Skills and Experience

• Certifications in Azure and other relevant technologies are highly beneficial.
• At least 5 years of experience in data engineering, managing infrastructure in both on-premises and cloud settings.

This role requires active SC clearance and willingness to undergo Developed Vetting (DV). The position also involves working on-site 5 days a week at locations around Worcester, Gloucester, Great Malvern, Ebbw Vale, and Brynmawr. If you are passionate about data and technology in hybrid environments, we encourage you to apply.