Data Engineer

Carnall Farrar; Greater London; On-site; GBP 55,000 - 75,000; Full time

Benefits offered by this job

Hybrid working

Job summary

Carnall Farrar is seeking a Data Engineer to join the Data Innovation team, working on the CFlakehouse Databricks platform. You will write SQL and Python, build and maintain automated pipelines, and convert raw healthcare data into structured, production-ready material for consultants and clients.

You'll learn from senior engineers, contribute to technical products, and expand your scope as you grow in a healthcare data consultancy that values impact and delivery.

Qualifications

• Around 2 to 3 years of experience in a data engineering or comparable technical data role.
• Working SQL, used day-to-day on the lakehouse.
• Intermediate Python for data transformation and automation.
• Experience building automated data pipelines using PySpark or Spark, with appropriate monitoring.
• Ability to write unit, functional and integration tests.
• Comfort working to a Git and GitHub workflow.
• Curiosity about healthcare data and how it relates to care delivery and policy.
• A clear communicator who knows when to ask for support and works well in a team.

Responsibilities

• Build and maintain automated data pipelines using PySpark and Spark, with appropriate monitoring.
• Build data models and pipelines under the guidance of senior colleagues, producing production-ready code or client-ready material.
• Develop data quality, validation and consistency checks, and carry out data cleansing.
• Develop unit, functional and integration tests, and follow the team’s Git and GitHub workflow.
• Participate in agile ways of working: keep user stories and tasks up to date, and contribute to stand-ups, retros and show and tells.
• Flag early when work is deviating from plan, and help identify and deliver mitigations.
• Communicate technical solutions to non-technical colleagues, and give and receive feedback well.

Skills

SQL

Python

PySpark

Spark

Data Pipelines

Unit tests

Git/GitHub

Data Modeling

Education

2-3 years experience

Tools

Databricks Lakehouse

GitHub workflows

Job description

About us

We are a leading consultancy with a purpose to make an enduring impact on health and healthcare. We work with leaders and frontline teams to improve health, transform healthcare, drive adoption of innovation and create value through investment.

Our consultancy serves the entire healthcare sector,from payors and providers of care,to life science companies, health tech and sector suppliers and health investors. We provide end-to-end services, from strategy through implementation, accelerated by data,digitaland AI.

We shape opinion through evidence-based thought leadership on key issues affecting health. With unmatched ability to access and use health data, our consultants are a driving force for delivering positive and meaningful change.

About the role

The Data Engineer sits within the Data Innovation team and works day-to-day on the CFlakehouse, our Databricks platform that holds the routine healthcare data behind our analytical and product work. The role is hands-on and delivery-focused: writing SQL and Python,buildingandmaintainingautomated pipelines, and turning raw data into structured, production-ready material that consultants and clients can rely on.

CF runs an apprentice model of development. You will learn actively from senior engineers, contribute to the team’s collective knowledge, and take on greater scope as you grow. The work spans data cleansing and validation for client engagements, building andmaintainingpipelines, and contributing to CF’s technical products, including through hackathons.

This is a good fit for an engineer with a couple of years of experience who wants to build a broad technical foundation in a data-rich healthcare consultancy, work close to real client problems, and see how technical work turns into client value.

Responsibilities

The requirements,responsibilitiesand duties of the role will include, but are not limited to:

• Build andmaintainautomated data pipelines usingPySparkand Spark, withappropriate monitoring
• Build data models and pipelines under the guidance of senior colleagues, producing production-ready code or client-ready material
• Develop data quality,validationand consistency checks, and carry out data cleansing
• Develop unit, functional and integration tests, and follow the team’s Git and GitHub workflow
• Participate in agile ways of working: keep user stories and tasks up to date, and contribute to stand-ups, retros and show and tells
• Flag early when work is deviating from plan, and helpidentifyand deliver mitigations

Domain,clientsand collaboration

• Build an understanding of healthcare data and the CFlakehouse: what is collected, how it is structured, and how new sources are brought in
• Work with teams to bring analytical insights to client problems
• Communicate technical solutions to non-technical colleagues, and give and receive feedback well

Learning and contribution

• Learn best-practice development processes from senior colleagues, and seek out new tools and techniques to apply

• Use AI tools to improve code quality and efficiency

• Support business development and bid writing on technical detail, and contribute to product development through hackathons and thought leadership

The requirements of the role include:

• Around 2 to 3 years of experience in a data engineering or comparable technical data role

• Working SQL, used day-to-day on thelakehouse

• Intermediate Python for data transformation and automation

• Experience building automated data pipelines usingPySparkor Spark, withappropriate monitoring

• Ability to write unit, functional and integration tests

• Comfort working to a Git and GitHub workflow

• Curiosity about healthcare data and how it relates to care delivery and policy

• A clear communicator who knows when to ask for support and works well in a team

Desirable, and not expected on day one:

• Databrickslakehouseexperience: notebooks, Workflows and Jobs, Delta Lake

• UnityCatalogbasics (we will train)

• Awareness of cloud basics such as storage and access control, and of medallion structure, bronze to silver to gold

• Exposure todbtand basic dimensional modelling

Flexible working

We follow a hybrid working model that balances in person connections and remote work to drive exceptional client impact. We enjoy working in person together withclients and colleaguesand work where clients need us to be.

Insupporting flexibility and remote working, team members can work from home one day per week as standard. Additionally, we offer 44 remote working days per year which can be used to top up your working from home days and enable you to work from home up to two days per week-subject to client needs. Alternatively, you could use your allowance in blocks to manage school holidays or other commitments. Our core in person working hoursarefrom 10am until 4pm allowing you that extra flexibility to manage your schedule in a way that works for you.

Our commitment to Diversity & Inclusion

We are committed to building an inclusive and supportive culture where diversity thrives, and all our people can excel. We only recruit, promote and reward our people based on their skills and contribution, without regard to gender, race, disability, religion, nationality, ethnicity, sexual orientation, age, marital status, or other characteristics.

We are Disability Confident Accredited, and we want you to feel comfortable and able to perform at your best in the recruitment process. Ifyourequireany reasonable adjustments for any part of the recruitment process, please let us know.

• Holiday entitlement: 25 days/year for staff and 30 days/year for leadership, increasing by 1 day for every year of service up to a maximum of 35 days of holiday per year

• We contribute 7% of your salary into your pension, while you contribute 3% (or more if you like)

• Access to a flexible benefits programme giving you the chance to increase pension contributions, gain access to a cash plan or benefit from aClassPasssubscription

• Annual leave purchase: employees with less than 35 days annual leave entitlementare able topurchaseadditionalannual leave days

• Income protection:in the event oflong-term incapacity and a qualifying claim, 75% of salary will be paid

• Enhanced sick pay benefit beyond Statutory Sick Pay for up to a total 12 weeks in any 12-month period

• Life insurance covering four times your basic salary in a tax-free lump sum payable to your beneficiariesin the event ofyour death whilst in service

• Enhanced family leave policies:additionalpay for parents who have a baby or adopt

• Access to an interest-free loan of up to £10,000

• Access to an interest-free season ticket loan, repayable by 12 monthly instalments

• Workplace nursery scheme: access to a scheme to help working parents save tax and NI on the cost of the nursery care

• Flexible working policy: including the ability to work from homeup totwo days per week, with 44 remote working days peryear

• An employeeassistanceand wellness programme: including access to telephone counselling, life coaching, interactive tools online and digital content downloadable from Lifeworks

• Seasonal flu jabs: provided by Boots annually

• Eye care tests: vouchers and discounts at Vision Express

• Ride to work scheme, saving up to 42% on bikes and cycling accessories at Evans Cycles

• Membership to the Health Service Journal (HSJ)