Databricks Lead Engineer - nttlimited

OpenTalent; Greater London; On-site; GBP 70,000 - 110,000; Full time

Job summary

NTT DATA in London seeks an experienced Data Engineer/Architect to lead client engagements and delivery of data pipelines on Databricks and cloud-native platforms.

You will work across the data lifecycle—ingestion, transformation, modelling, governance, and consumption—hands-on with PySpark/SQL, and mentor junior engineers to build scalable data solutions.

Qualifications

• Minimum 5–8 years in data engineering, data warehousing, or data architecture roles with 3+ years on Databricks.
• University degree required; advanced degrees preferred in CS or data.
• Databricks certifications (Data Engineer) highly desirable.

Responsibilities

• Client Engagement & Delivery across data projects.
• Data Pipeline Development (Batch and Streaming) on Databricks and cloud-native platforms.
• Databricks & Lakehouse Architectures implementation.
• Data Modelling & Optimisation (Delta Lake, Medallion architecture).
• Collaboration & Best Practices across teams.
• Quality, Governance & Security compliance.

Skills

Data engineering

Client engagement

Mentoring teams

SQL

Python

Data governance

Problem solving

Education

University degree

BSc/MSc in Computer Science or Data Engineering

Tools

Databricks

DBT

Matillion

Talend

GitHub

CI/CD pipelines

Job description

Make an impact with NTT DATA Join a company that is pushing the boundaries of what is possible. We are renowned for our technical excellence and leading innovations, and for making a difference to our clients and society. Our workplace embraces diversity and inclusion – it’s a place where you can grow, belong and thrive.

Your day at NTT DATA

Primary Responsibilities

• Client Engagement & Delivery
• Data Pipeline Development (Batch and Streaming)
• Databricks & Lakehouse Architectures
• Data Modelling & Optimisation ( Delta Lake , Medallion architecture )
• Collaboration & Best Practices
• Quality, Governance & Security

Business Relationships

• Solution Architects
• Data Engineers, Developers, ML Engineers, and Analysts
• Client stakeholders up to Head of Data Engineering, Chief Data Architect, and Analytics leadership

To thrive in this role, you need to have:

Key Competencies

• Proven experience in data engineering and pipeline development on Databricks and cloud-native platforms.
• Strong consulting values with ability to collaborate effectively in client-facing environments.
• Hands-on expertise across the data lifecycle: ingestion, transformation, modelling, governance, and consumption.
• Strong problem-solving, analytical, and communication skills.
• Experience leading or mentoring teams of engineers to deliver high-quality scalable data solutions.

Technical Expertise

• Deep expertise with the Databricks platform ( Spark / PySpark /Scala, Delta Lake , Unity Catalog , MLflow ).
• Proficiency in ETL/ELT tools such as DBT , Matillion , Talend , or equivalent.
• Strong SQL and Python (or equivalent language) skills for data manipulation and automation.
• Hands-on experience with cloud platforms ( AWS , Azure , GCP ).
• Familiarity with Databricks Workflows and other orchestration tools.
• Knowledge of data modelling methodologies (star schemas, Data Vault, Kimball, Inmon).
• Familiarity with medallion architectures, data lakehouse principles and distributed data processing .
• Experience with version control tools ( GitHub , Bitbucket ) and CI/CD pipelines.
• Understanding of data governance , security, and compliance frameworks.
• Exposure to AI/ML workloads desirable.

Experience, Qualifications, and Education

• Experience: Minimum 5–8 years in data engineering, data warehousing, or data architecture roles, with at least 3+ years working with Databricks .
• Education: University degree required.
• Preferred: BSc/MSc in Computer Science, Data Engineering, or related field
• Databricks certifications (Data Engineer Professional) highly desirable.