Data Engineer

Whitehall Resources Limited; Cambridge; On-site; GBP 55,000 - 85,000; Full time

Job summary

Whitehall Resources Limited in Cambridge is seeking a Data Engineer for an initial 6 month contract, on-site two days per week. The role focuses on designing and delivering high-quality data solutions in Databricks, building and optimising ETL pipelines from systems such as ServiceNow, JIRA and Dynatrace to support reporting and analytics.

You will design and maintain efficient data pipelines, implement data modelling and schema design, and perform data quality validation to ensure trustworthy

Qualifications

• Minimum 4 years of experience building ETL pipelines in Databricks.
• Strong SQL and Python for data transformation.
• Experience with data modelling and schema design.
• Knowledge of data governance and data lineage practices.
• Experience with Git-based version control and CI/CD.

Responsibilities

• Design, build, and maintain scalable ETL pipelines in Databricks.
• Integrate data from ServiceNow, JIRA, and Dynatrace.
• Implement data validation and quality checks for Power BI reporting.
• Collaborate with Staff Data Engineer and Visualisation Developer.
• Manage code in Git and support CI/CD for Databricks deployments.
• Contribute to data lineage documentation and governance standards.

Skills

Databricks

SQL

Python

Data modelling

Schema design

Data quality

Git

CI/CD

Data governance

Data lineage

Agile

Tools

Databricks

Git

ServiceNow

JIRA

Dynatrace

Azure DevOps (ADO)

Job description

Whitehall Resources are currently looking for a Data Engineer for an initial 6 month contract.

*** INSIDE IR35 ***

*This role will require on site work in Cambridge 2 days per week.

Job Spec:

Our client are seeking a motivated and detail-oriented Data Engineer with a passion for designing and delivering high-quality data solutions in Databricks.

You will be responsible for building and optimising data pipelines that form the foundation of our reporting and analytics ecosystem.

Using your technical expertise, you will design and maintain efficient ETL processes to integrate data from systems such as ServiceNow, JIRA, and Dynatrace, ensuring accuracy, performance, and scalability.

• Main Responsibilities:Design, build, and maintain scalable ETL pipelines in Databricks to integrate data from multiple business systems such as ServiceNow, JIRA, and ADO.
• Optimise data workflows for performance, scalability, and reliability.
• Implement data validation and quality checks to ensure trustworthy reporting in downstream tools such as Power BI.
• Create data modelling and schema design at backend for analytics and reporting.
• Collaborate with the Staff Data Engineer and Visualisation Developer to align technical delivery with reporting needs.
• Manage code in Git and support CI/CD processes for Databricks deployments.
• Contribute to data lineage documentation, standards, and governance best practices.
• Key Skills and Experience:Proven experience (minimum 4 years) in building and maintaining ETL pipelines using Databricks.
• Strong knowledge of SQL and Python for data transformation and automation.
• Proven experience in data modelling, schema design, and data quality validation.
• Solid understanding of data performance tuning and pipeline optimisation in Databricks.
• Experience working with Git-based version control and collaborative development workflows.
• Strong analytical and problem-solving skills, with an eye for efficiency and accuracy.
• Excellent communication and collaboration skills, comfortable working with both technical and non-technical partners.
• Desirable Skills:Familiarity with Agile delivery methods and iterative development practices
• Knowledge of data governance and data lineage documentation standards.
• Exposure to automation and CI/CD frameworks within Databricks.