Databricks Architect: Data Mesh & Real-Time Pipelines

Infosys; Greater London; On-site; GBP 110,000 - 150,000; Full time

Job summary

Infosys in London seeks a Senior Databricks Architect to design and operate scalable data pipelines and products. You will integrate legacy mainframe sources with cloud-native platforms to enable analytics and domain-owned data products.

The role focuses on Databricks, Spark, PySpark/SQL, and real-time ingestion using Kafka within an AWS-based data lake environment. Collaboration with multiple teams is essential.

Qualifications

• Strong experience with Databricks (Spark, PySpark/SQL) and data lake architectures.
• Hands-on experience with streaming technologies, especially Kafka / Confluent Kafka.
• Experience integrating legacy data sources such as DB2 (mainframe) and SQL Server.
• Working knowledge of AWS data services, especially S3.
• Understanding of data product thinking, data mesh concepts, and modern data engineering best practices.

Responsibilities

• Build and maintain batch and streaming data pipelines from legacy mainframe DB2 and on prem SQL Server systems into an AWS-based data lake.
• Implement real-time ingestion using Confluent Kafka and land data in Amazon S3.
• Develop data transformations and analytics pipelines in Databricks using the medallion architecture (Bronze, Silver, Gold).
• Design and deliver high-quality, reusable data products aligned to data mesh principles.
• Ensure data quality, reliability, security, and governance across pipelines.
• Collaborate with domain teams, platform teams, and stakeholders to enable self-serve analytics and consumption.

Skills

Databricks (Spark PySpark/SQL)

Kafka / Confluent Kafka

DB2 mainframe

SQL Server

AWS S3

Data mesh concepts

Data engineering best practices

Tools

Databricks

Confluent Kafka

DB2

SQL Server

AWS S3

Job description

Infosys in London seeks a Senior Databricks Architect to design and operate scalable data pipelines and products. You will integrate legacy mainframe sources with cloud-native platforms to enable analytics and domain-owned data products.

The role focuses on Databricks, Spark, PySpark/SQL, and real-time ingestion using Kafka within an AWS-based data lake environment. Collaboration with multiple teams is essential.