Senior Data Engineer

TN United Kingdom; Malvern Hills; On-site; GBP 50,000 - 90,000; Full time

Job summary

An established industry player is seeking a Senior Data Engineer to enhance their data infrastructure across on-premises and Azure environments. This role is pivotal in designing and maintaining data systems that drive innovation and support advanced analytics. You will work collaboratively to develop data pipelines, manage real-time data streams, and ensure data integrity while mentoring team members. This position offers a unique opportunity to contribute to impactful projects and foster a data-centric culture within the organization. If you are passionate about leveraging data to drive decision-making, this role is for you.

Benefits

Autonomy to develop skills

Participation in impactful projects

Development opportunities

Wellness support

Flexible working

Generous leave

Volunteering days

Pension

Life insurance

Medical and travel insurance

Qualifications

• 5+ years in data engineering with hybrid environments.
• Strong Python skills for automation and scripting.

Responsibilities

• Develop and manage data pipelines using Python.
• Implement and manage hybrid cloud and data storage solutions.

Skills

Python

ETL/ELT workflows

Hybrid cloud data architecture

Docker

Kubernetes

Apache Airflow

Kafka

Security practices

Elasticsearch

PostgreSQL

Education

Relevant certifications in Azure

5+ years in data engineering

Tools

Docker

Kubernetes

Apache Airflow

Kafka

Elasticsearch

PostgreSQL

Job description

Social network you want to login/join with:

Methods Analytics (MA) exists to improve society by helping people make better decisions with data. Combining passionate people, sector-specific insight, and technical excellence to provide our customers an end-to-end data service.

We use a collaborative, creative, and user-centric approach to data to do good and solve difficult problems. Ensuring that our outputs are transparent, robust, and foster discussion and debate as part of our approach. We will question assumptions, ambition, and process – but do so with respect and humility.

We relish difficult problems and overcome them with innovation, creativity, and technical freedom to design optimal solutions. Ethics, privacy, and quality are at the heart of our work, and we will not sacrifice these for outcomes.

We treat data with respect and use it only for the right purpose. Our people are positive, dedicated, and relentless. Data is a vast topic, but we strive for interactions that are engaging, informative, and fun in equal measure, while maintaining a steely focus on outcomes and delivering quality products for our customers.

We are passionate about our people; we want our colleagues to develop their strengths and enjoy their work.

Methods was acquired by the Alten Group in early 2022.

Requirements

On-site, Full-time.

We are seeking a seasoned Senior Data Engineer (Infrastructure) to join our team. This role is essential for designing, building, and maintaining sophisticated data infrastructure systems that operate across on-premises and Azure cloud environments. The position involves deploying and managing scalable data operations that support advanced analytics and data-driven decision-making, crucial for our organisational growth and innovation.

Responsibilities

• Develop and manage data pipelines using Python, supporting streaming and batch processing in structured, semi-structured, and unstructured data across on-premises and Azure environments.
• Implement and manage hybrid cloud and data storage solutions, ensuring seamless data integration and accessibility.
• Utilise Docker for containerisation and Kubernetes for orchestration to ensure scalable deployment.
• Automate workflows with tools like Apache Airflow.
• Manage real-time data streams using event-driven technologies such as Kafka.
• Manage security and access controls, incorporating tools like Keycloak to ensure data integrity and compliance.
• Oversee Elasticsearch setups for data search and analytics.
• Administer and optimise PostgreSQL databases for high performance and availability.

Essential Skills and Experience

• Strong Python skills for scripting and automation.
• Experience with ETL/ELT workflows in hybrid environments.
• Profound knowledge of hybrid cloud data architecture, especially Azure.
• Experience with Docker and Kubernetes.
• Proficiency with Apache Airflow.
• Experience with Kafka and real-time data streaming.
• Knowledge of security practices including Keycloak.
• Management of Elasticsearch and PostgreSQL in hybrid setups.

Your Impact

You will enable data-driven decision-making by delivering accurate, timely insights from a hybrid infrastructure, driving scalability and flexibility. Your work will foster a data-centric culture, mentoring team members and advancing engineering practices.

Desirable Skills and Experience

• Certifications in Azure and relevant technologies.
• Minimum of 5 years in data engineering, managing infrastructure in hybrid environments.

This role requires active SC clearance and willingness to undergo Developed Vetting (DV). The position demands on-site presence 5 days a week at our locations in Worcester, Gloucester, Great Malvern (England), Ebbw Vale, and Brynmawr (Wales). If passionate about using data to drive innovation in hybrid environments, we encourage you to apply.

Benefits

• Autonomy to develop skills and experience.
• Participation in impactful projects.
• Strong leadership and collaborative environment.
• Development opportunities including LinkedIn Learning and training.
• Wellness support, flexible working, social activities, generous leave, volunteering days, pension, bonuses, life insurance, medical and travel insurance, enhanced parental pay, travel schemes, and more.