Senior Data Engineer - Databricks Platform Lead

TD; Belfast City District; On-site; GBP 90,000 - 120,000; Full time

Job summary

TD in Belfast is seeking a Senior Data Engineer to build and evolve the strategic Research data platform on Databricks. You will design scalable data pipelines and governed data products, enabling analytics, BI reporting, and AI initiatives.

The role requires hands-on engineering and leadership to coordinate with global stakeholders and offshore teams while delivering production-grade data platforms.

Qualifications

• 10+ years of experience in data engineering, data platform delivery, software engineering, or related enterprise technology roles.
• Databricks, Spark / PySpark, Python, SQL, Delta Lake, Azure, ADLS, and cloud-based data engineering patterns.
• Design scalable data pipelines and lakehouse solutions using modern cloud data platform patterns.
• Unity Catalog or similar governance capabilities (data discovery, access control, lineage, ownership, catalog structures).
• Deliver enterprise-grade data platforms including CI/CD, monitoring, data quality, operational controls, and support processes.
• Strong analytical and problem-solving skills across complex data, platform, and integration challenges.
• Excellent communication to work across local, global, and offshore teams.
• Coordinate with India-based delivery resources to bridge time zones and execution gaps.
• Exposure to AI-ready data platform concepts (metadata enrichment, unstructured data processing, AI enablement).
• Knowledge of Research, Capital Markets, Global Markets, client analytics, readership, CRM, revenue, or market data domains would be beneficial.

Responsibilities

• Work as part of a Belfast-based data engineering team contributing to globally used Research data platforms, products, and services.
• Build the foundational Research data platform that will serve as a strategic data and AI enabler for the Research business.
• Design, develop, and maintain scalable data pipelines using Python, SQL, Spark, Databricks, Azure, and related cloud data technologies.
• Create curated, reusable, and governed data products that can be consumed by Research applications, dashboards, reporting tools, and AI-enabled solutions.
• Support data ingestion and transformation patterns across structured, semi-structured, and unstructured data sources where required.
• Contribute to data domain, table, object, field, and metadata structures that align with Unity Catalog and broader data governance patterns.
• Implement data quality, reconciliation, lineage, monitoring, audit, and operational controls required for production-grade data platforms.
• Support CI/CD, automation, deployment, and environment management practices for Databricks and Azure-based data workloads.
• Partner with global engineering colleagues, business stakeholders, platform teams, and data consumers to translate requirements into scalable data solutions.
• Act as coordination bridge with Cognizant resources in India, using Belfast time zone overlap to support delivery alignment and execution.

Skills

Databricks

Spark / PySpark

Python

SQL

Delta Lake

Azure

Unity Catalog

Data governance

CI/CD

Data quality

Job description

TD in Belfast is seeking a Senior Data Engineer to build and evolve the strategic Research data platform on Databricks. You will design scalable data pipelines and governed data products, enabling analytics, BI reporting, and AI initiatives.

The role requires hands-on engineering and leadership to coordinate with global stakeholders and offshore teams while delivering production-grade data platforms.