Senior Data Engineer (DATABRICKS)

Mastercard; Shefford; On-site; GBP 70,000 - 90,000; Full time

Job summary

A leading financial services provider in the UK is seeking a Senior Data Engineer specializing in Databricks administration on AWS. You will be responsible for setting up, configuring, and optimizing the Databricks platform. The ideal candidate has over 6 years of experience, a deep understanding of Databricks architecture, and experience with IAM policies. Join us to help empower businesses and create innovative solutions in the digital economy.

Qualifications

• 6+ years of experience in Databricks administration on AWS or multi-cloud environments.
• Deep understanding of Databricks workspace architecture and cluster configuration best practices.
• Strong experience in managing IAM policies and SCIM integration.

Responsibilities

• Provision and manage Databricks Workspaces across multiple environments.
• Implement and enforce role-based access controls and ensure compliance.
• Monitor workspace performance and provide L2/L3 support.

Skills

Databricks administration

AWS

Infrastructure as Code (Terraform)

IAM Policies

ETL orchestration

Tools

GitHub

CloudWatch

GitLab

Okta

Azure/AWS SSO

Job description

Our Purpose

Mastercard powers economies and empowers people in 200+ countries and territories worldwide. Together with our customers, we’re helping build a sustainable economy where everyone can prosper. We support a wide range of digital payments choices, making transactions secure, simple, smart and accessible. Our technology and innovation, partnerships and networks combine to deliver a unique set of products and services that help people, businesses and governments realize their greatest potential.

Title and Summary

Senior Data Engineer (DATABRICKS) Role Overview

As a Senior Databricks Administrator, you will be responsible for the setup, configuration, administration, and optimization of the Databricks Platform on AWS. This role will play a critical part in managing secure, scalable, and high-performing Databricks environments, with a strong focus on governance, user access management, cost optimization, and platform operations. You will collaborate closely with engineering, infrastructure, and compliance teams to ensure that the Databricks platform meets enterprise data and regulatory requirements.

Key Responsibilities

1. Databricks Platform Administration (AWS)
   • Provision, configure, and manage Databricks Workspaces across multiple environments (Dev, UAT, Prod).
   • Define and enforce cluster policies, compute tagging, and usage controls to optimize cost and resource utilization.
   • Administer Unity Catalog, including creation and management of catalogs, schemas, access control, and data lineage.
   • Manage SCIM integrations for user/group provisioning with IdPs like Okta or Azure/AWS SSO.
   • Configure and maintain CI/CD pipelines for notebook deployment and workflow promotion using GitHub, GitLab, or similar tools.
2. Security, Governance & Compliance
   • Implement and enforce role-based access controls (RBAC) and fine-grained data permissions using Unity Catalog and Lake Formation.
   • Ensure auditability and lineage tracking for compliance with Open Banking, GDPR, and PSD2 regulations.
   • Configure and manage token scopes, secrets management, and credential passthrough for secure access to underlying data.
   • Work with InfoSec and compliance teams to ensure platform security aligns with regulatory frameworks.
3. Monitoring, Support & Troubleshooting
   • Monitor workspace performance, cluster health, job execution, and user activity using Databricks native tools, CloudWatch, and third-party integrations.
   • Provide L2/L3 support for Databricks usage issues, including notebook errors, job failures, and workspace anomalies.
   • Maintain operational runbooks, automation scripts, and alerting mechanisms for platform health and governance events.
4. Automation & Best Practices
   • Build and manage Terraform/IaC scripts for environment provisioning and infrastructure consistency.
   • Define naming conventions, resource tagging standards, and workspace governance guidelines.
   • Promote standardization of reusable notebook templates, shared cluster configurations, and workflow orchestrations.
   • Drive internal knowledge sharing, onboarding, and platform enablement sessions.

Must-have Skills

• 6+ years of experience in Databricks administration on AWS or multi-cloud environments.
• Deep understanding of Databricks workspace architecture, Unity Catalog, and cluster configuration best practices.
• Strong experience in managing IAM policies, SCIM integration, and access provisioning workflows.
• Hands-on experience with monitoring, cost optimization, and governance of large-scale Databricks deployments.
• Hands-on experience with infrastructure-as-code (Terraform) and CI/CD pipelines.
• Experience with ETL orchestration and collaboration with engineering teams (Databricks Jobs, Workflows, Airflow).

Good-to-Have Skills

• Understanding of Delta Lake, Lakehouse architecture, and data governance patterns.
• Experience in AWS Glue and database technologies SQL (Amazon Aurora) and NoSQL (MongoDB/DynamoDB).
• Exposure to regulated industries such as banking, fintech, or healthcare.
• Experience with incident response, audit logging, and security remediation.

Preferred Certifications

• Databricks Certified Data Engineer Associate/Professional
• Databricks Certified Lakehouse Platform Administrator
• AWS Certified Solutions Architect – Associate

Corporate Security Responsibility

All activities involving access to Mastercard assets, information, and networks comes with an inherent risk to the organization and, therefore, it is expected that every person working for, or on behalf of, Mastercard is responsible for information security and must:

• Abide by Mastercard’s security policies and practices;
• Ensure the confidentiality and integrity of the information being accessed;
• Report any suspected information security violation or breach;
• Complete all periodic mandatory security trainings in accordance with Mastercard’s guidelines.