Senior Data Engineer

Problemshared; United Kingdom; Remote; GBP 55,000 - 80,000; Full time

Job summary

A remote digital mind health provider is seeking a Senior Data Engineer to ensure data accessibility and accuracy. You will design ingestion pipelines and collaborate closely with the Analytics Engineer. Ideal candidates have extensive experience with data frameworks and AWS, alongside strong programming skills in Python or R. The role offers a flexible work environment and various employee benefits.

Benefits

30 days annual leave plus public holidays

Private health insurance

Pension scheme

Flexible working options

Access to wellbeing activities

Development and training opportunities

Qualifications

• Proven experience with ingestion frameworks like Fivetran and analytical databases/lakes.
• Production experience working in AWS with infrastructure as code.
• Strong Python and/or R skills, and comfort with non-relational data formats.

Responsibilities

• Design, deploy and monitor Fivetran connectors for various data sources.
• Develop Databricks jobs to preprocess raw data.
• Implement AWS-integrated authentication and access controls.

Skills

Ingestion frameworks experience

Analytical databases / lakes experience

AWS experience

Python skills

R skills

SQL fluency

Education

STEM degree

Tools

Databricks

Terraform

Azure DevOps

GitHub Actions

Job description

Overview

Please be aware although we are a remote organisation we do require candidates to reside in the UK.

Salary: Competitive

Work pattern: This is a permanent role, working 40 hours per week Monday to Friday.

Reporting to: Head of Analytics

About the Job

ProblemShared has a newly formed Data Office, and we’re looking for a Senior Data Engineer to work closely with our Senior Analytics Engineer, ensuring our data is accessible, accurate and actionable.

We believe in “data for good” - using insights to help neurodivergent people better understand themselves, enable practitioners to deliver more effective care, and guide teams in improving support. We're building a modern data platform (Fivetran → Databricks + DBT → PowerBI) that integrates data from clinical, operational and corporate systems into a safe, accessible, and reliable model, enabling real-time insight and continuous improvement.

At ProblemShared, we blend people, data and technology to support individuals on their mental health or neurodevelopmental journey - and we believe you can do the most meaningful work of your career here.

About Us

We are a practitioner-led, CQC-regulated, digital mind health provider, working with a community of expert practitioners to broaden access to the highest quality care for people across the UK. By leveraging technology, we aim to provide scalable and personalized solutions that enhance mental well-being and bridge gaps in mental healthcare delivery.

We work in partnership with institutions such as the NHS, insurance companies and universities to deliver therapy, psychiatry, neurodevelopmental assessments and post diagnostic care for adults, children and young people, mainly around Autism and ADHD. Key to our mission is the innovative and effective use of data analytics and resulting insights.

The Role

You own the “on-ramp” for all our operational data—building and maintaining ingestion pipelines from our AWS-hosted Postgres system, Zendesk, billing feeds, and more into Databricks. While your primary focus is ingestion, you’ll work hand-in-glove with the Analytics Engineer to understand downstream needs, help troubleshoot transformations, and step in to assist with Silver-layer tasks when peers are out. You’ll also implement authentication, anonymisation and any non-SQL transformations needed upstream of our analytic models.

What you’ll do

• Design, deploy and monitor Fivetran (or equivalent) connectors for PostgreSQL, Zendesk, Semble, Bob, and other API and flat-file exports.
• Develop Databricks jobs and notebooks to preprocess raw data: validation, pseudonymisation, format conversions (JSON/Parquet).
• Implement and maintain AWS-integrated authentication and role-based access controls in Databricks. (NB. User authentication in ProblemShared uses Microsoft, but our core internal tech stack is on AWS)
• Write scripts (Python or R) to handle edge-case formats and automate testing/alerting.
• Collaborate daily with the Analytics Engineer—providing context on schema changes, edge cases and PII nuances.
• Cross-train so you can assist with Silver-layer cleanup or Gold-layer hand-offs during leave or as part of some spikes of work.
• Partner closely with Software Engineers in our core system teams.
• Continuously evaluate and bring in new ingestion or analytical tools (e.g. DuckDB, LLMs) where it adds speed, resilience or adaptability.

What you will bring to the role

Must have

• Proven experience with a variety of ingestion frameworks (e.g. Fivetran) and analytical databases / lakes (e.g. Redshift, Snowflake, Databricks).
• Production experience working in AWS and infrastructure as code (Terraform, Azure DevOps or GitHub Actions) in an established data team.
• Hands-on with Databricks on AWS: jobs, notebooks, IAM integration.
• Strong Python and/or R skills, and comfort working with non-relational data formats.
• Strong SQL fluency against Postgres and cloud stores.
• Action orientation: we value thoughtful, reliable solutions built to deliver immediate value.

Nice to have

• STEM degree (Computer Science, Engineering, Maths, Stats, Physics or equivalent)
• Knowledge of PII controls: encryption, masking, pseudonymisation and role-based access.
• Experience with JSON/Parquet/HL7 or event-driven tools (Airflow, Kafka).
• AWS certification (e.g. Data Analytics – Specialty).
• Basic Linux administration or Docker.

What makes us different

• Fully remote, flexible: Work where you do your best thinking.
• Neurodiversity-welcoming: We encourage applications from autistic, ADHD and other neurodivergent candidates.
• Tool-agnostic, outcome-driven: We value mental agility and diverse tool experience (Snowflake, Redshift, DuckDB, Databricks, Python, R…) over narrow specialism.
• Inclusive by design: We focus on potential over pedigree, and adapt our process to support your needs. We also know that everyone has a life outside work, so we’re happy to discuss flexible working. We embrace difference and individuality and are proud to be equal opportunity employer.

What else do we offer you

• 30 days annual leave + public holidays + the option to buy and sell additional leave & extended leave options such as sabbatical leave
• Private health insurance and pension scheme
• Enhanced family friendly policies
• Flexible working with the option of free co-working
• All company and team in person meet ups
• Access to a range of wellbeing activities
• Access to development / training opportunities to support your career ambitions
• One volunteering day a year

Our Recruitment Process and Next Steps

At ProblemShared, we see beyond just your CV. We're genuinely excited about who you are, your potential, and the unique qualities you bring to our team. We also recognise that life outside of work is important, which is why we prioritise being a flexible employer.

We're proud to be an equal opportunity and Disability Confident Committed employer and celebrate diversity and are ambitious in creating a recruitment process that allows everyone to shine and showcase their unique qualities. We offer tailored opportunities and adjustments to help each individual reach their full potential.

Screening

Once we review your CV a member of our recruitment team will be in touch to learn more about your experience and motivations.

Technical Interview

The next step is to meet with our hiring team who will take you through an interview to allow you to demonstrate your ability and suitability for the role. If successful we will move to an offer of employment and pre-employment checks.

We know that applying for a new role and considering a new team to join can be a daunting process. If you have never worked remotely before, this can also generate a lot of questions regarding the environment and ways of working. We aim to make your experience with us as informative and enjoyable as possible.

Please don't hesitate to apply - we would love to hear from you!