Data Engineer: Real-Time Pipelines on Databricks

Searchability®; Greater Manchester; Hybrid; GBP 54,000 - 66,000; Full time

Benefits offered by this job

Discretionary bonus

Share option plan

Pension scheme

Life insurance

Health plan

Holiday entitlement

Study support

Employee assistance 24/7

Office social events

Job summary

Talantir is hiring a Data Engineer to design and optimise scalable data pipelines in a Databricks Lakehouse. You will work with streaming and batch data using Spark Structured Streaming and Kafka (MSK), ensuring efficient data flows and governance across analytics, reporting, and operations.

Your role includes CI/CD for data workflows and close collaboration with BI, product, and engineering teams. The position offers a hybrid working model in a collaborative office environment with a

Qualifications

• Strong experience with Databricks and Spark (PySpark or Scala).
• Proven capability building streaming and batch pipelines using Spark Structured Streaming and Kafka (MSK).
• Solid AWS experience (S3, Glue, Lambda, Batch, IAM) and data modelling skills.

Responsibilities

• Design, build, and optimise scalable data pipelines within a Databricks Lakehouse environment.
• Develop streaming and batch pipelines using Spark Structured Streaming and Kafka; ensure data quality and governance.
• Collaborate with BI, product, and engineering to deliver high-quality, governed data for analytics and reporting.

Skills

Databricks

Apache Spark

Python

SQL

PySpark

Scala

CI/CD

Git

AWS

Delta Lake

Delta Live Tables

Medallion Architecture

Spark Structured Streaming

Kafka

MSK

S3

Glue

Lambda

Batch processing

IAM

Tools

Databricks

Apache Spark

Kafka (MSK)

Delta Lake

Delta Live Tables

AWS

Python

SQL

CI/CD tools

Git

Job description

Talantir is hiring a Data Engineer to design and optimise scalable data pipelines in a Databricks Lakehouse. You will work with streaming and batch data using Spark Structured Streaming and Kafka (MSK), ensuring efficient data flows and governance across analytics, reporting, and operations.

Your role includes CI/CD for data workflows and close collaboration with BI, product, and engineering teams. The position offers a hybrid working model in a collaborative office environment with a