Senior Data Engineer

Response Informatics; Salford; On-site; GBP 45,000 - 70,000; Part time

Job summary

A leading IT consulting firm in Salford is seeking a Data Engineer to design and develop metadata-driven data pipelines using Azure Data Factory and Databricks. The ideal candidate will have expertise in optimizing data workflows with SparkSQL and Pandas, and experience with CI/CD for deployments. This role involves collaborating with cross-functional teams and ensuring data quality and governance. The position is contract-based and requires proficiency in Python or Scala.

Qualifications

• Expertise in building and orchestrating data pipelines using Azure Data Factory.
• Hands-on experience with Databricks notebooks and clusters.
• Advanced data manipulation skills using Pandas.
• Strong knowledge of distributed data processing with SparkSQL.
• Experience with CI/CD processes and tools like Azure DevOps.

Responsibilities

• Design, develop, and maintain metadata-driven data pipelines.
• Build and implement metadata frameworks, ensuring scalability.
• Optimize data workflows using SparkSQL and Pandas.
• Collaborate with teams to integrate data solutions into architecture.
• Implement CI/CD pipelines for automated deployment and testing.
• Ensure data quality and compliance with standards.
• Provide technical leadership and ownership of projects.

Skills

Azure Data Factory

Databricks

Pandas

SparkSQL

CI/CD with Azure DevOps

Metadata-driven architecture

Python

Scala

Tools

Azure Data Factory

Databricks

Pandas

SparkSQL

CI/CD

Metadata-driven architecture

Python/Scala

Job description

Responsibilities

• Design, develop, and maintain metadata-driven data pipelines using Azure Data Factory and Databricks.
• Build and implement end-to-end metadata frameworks, ensuring scalability and reusability.
• Optimize data workflows leveraging SparkSQL and Pandas for large-scale data processing.
• Collaborate with cross‑functional teams to integrate data solutions into enterprise architecture.
• Implement CI/CD pipelines for automated deployment and testing of data solutions.
• Ensure data quality, governance, and compliance with organizational standards.
• Provide technical leadership and take complete ownership of assigned projects.

Technical Skills Required

• Azure Data Factory (ADF): Expertise in building and orchestrating data pipelines.
• Databricks: Hands‑on experience with notebooks, clusters, and job scheduling.
• Pandas: Advanced data manipulation and transformation skills.
• SparkSQL: Strong knowledge of distributed data processing and query optimization.
• CI/CD: Experience with Azure DevOps, Git, or similar for automated deployments.
• Metadata-driven architecture: Proven experience designing and implementing metadata frameworks.
• Programming: Proficiency in Python and/or Scala for data engineering tasks.

Seniority level

• Associate

Employment type

• Contract

Job function

• Information Technology
• IT Services and IT Consulting

Referrals increase your chances of interviewing at Response Informatics by 2x.

Location: Salford, England, United Kingdom.