Senior Data Engineering Manager
Crisil
A complete application in a minute — tailored resume and cover letter, ready to send.
Job summary
Crilis is seeking a Senior Data Engineering Manager to lead the design, implementation, and evolution of a next-generation data platform on Databricks. The role combines hands-on engineering with senior leadership to drive platform migration, governance, and data capabilities across the organization.
You will guide a team of data engineers, define reference architectures, and partner with architects, product teams, and business stakeholders to ensure scalable, secure, and cost-efficient
Qualifications
- 8–12 years of data engineering experience with leadership.
- Proven experience architecting and delivering modern data platforms using Databricks.
- Strong leadership and stakeholder management across business and tech teams.
Responsibilities
- Own the technical direction of the Databricks data platform and establish engineering standards.
- Lead architecture and design decisions for enterprise-scale data products and platform capabilities.
- Mentor data engineers and promote engineering excellence.
- Drive platform strategy, governance, observability, and cost efficiency.
- Deliver data transformation initiatives and manage cross-workstream dependencies.
- Implement DevOps and DataOps standards including CI/CD practices.
Skills
Tools
Job description
We are seeking a Senior Data Engineering Manager to provide technical leadership for the design, implementation, and evolution of our next-generation data platform on Databricks.
As the Lead Data Engineer, you will be accountable for establishing engineering best practices, shaping architectural decisions, and driving the successful delivery of enterprise-scale data solutions. You will act as the technical authority for data engineering, partnering closely with Architects, Product teams, Technology teams, and business stakeholders to ensure the platform remains scalable, secure, cost-effective, and aligned to strategic business objectives.
You will combine hands‑on engineering expertise with technical leadership, guiding a team of data engineers through platform migration, solution design, and delivery excellence while helping transform the organisation's data capabilities.
Initially reporting to the Director of Data Transformation, you will be responsible for:
- Own the technical direction of the Databricks data platform and establish engineering standards across all data engineering initiatives.
- Lead architecture and design decisions for enterprise-scale data products and platform capabilities.
- Define reference architectures, reusable design patterns, and engineering guardrails for data ingestion, transformation, orchestration, governance, observability, and deployment.
- Act as the senior technical escalation point for complex engineering challenges.
- Lead, coach, and mentor a team of data engineers, fostering technical growth and engineering excellence.
- Drive engineering best practices through code reviews, technical design reviews, pair programming, and knowledge sharing.
- Promote a culture of quality, ownership, continuous improvement, and delivery excellence.
- Support recruitment, onboarding, capability development, and succession planning within the engineering team.
- Serve as the organisation's subject matter expert for Databricks and Lakehouse architecture.
- Drive adoption of Databricks capabilities including Delta Lake, Unity Catalog, Workflows, Lakeflow, SQL Warehouses, Serverless Compute, and emerging platform capabilities.
- Define the roadmap for platform optimisation, scalability, reliability, and cost efficiency.
- Partner with enterprise architecture to shape future‑state data platform strategy.
- Lead implementation of enterprise data governance standards using Unity Catalog and associated governance capabilities.
- Ensure secure, compliant, and auditable data access patterns across all environments.
- Define policies and standards for metadata management, lineage, data quality, and regulatory compliance.
- Lead technical delivery of strategic data transformation initiatives and migration programmes.
- Work closely with Product Managers, Architects, and business stakeholders to translate business outcomes into technical solutions.
- Manage technical dependencies, risks, and architectural decisions across multiple workstreams.
- Ensure solutions are delivered to agreed standards for performance, reliability, scalability, and maintainability.
- Establish and maintain DevOps and DataOps standards across the platform.
- Lead CI/CD adoption, Infrastructure-as-Code implementation, automated testing, release governance, and operational monitoring.
- Drive platform observability, performance optimisation, reliability engineering, and operational excellence.
- 8-12 years' experience in Data Engineering, including significant experience leading engineering teams and driving large-scale data platform initiatives.
- Demonstrated experience as a Lead Data Engineer, Technical Lead, Principal Engineer, or equivalent senior engineering position.
- Proven experience architecting and delivering modern data platforms using Databricks.
- Deep expertise in Apache Spark (PySpark, Spark SQL, Scala).
- Expert‑level knowledge of Delta Lake and Lakehouse architecture patterns.
- Advanced proficiency in Python and SQL.
- Strong experience with cloud‑native data platforms, ideally AWS.
- Proven expertise implementing Unity Catalog and enterprise governance frameworks.
- Experience with Infrastructure as Code using Terraform.
- Strong understanding of data warehousing, dimensional modelling, and data architecture principles.
- Experience implementing modern DataOps, DevOps, CI/CD, and automated testing practices.
- Ability to lead and influence engineering teams without compromising hands‑on technical credibility.
- Experience shaping engineering operating models, standards, and governance processes.
- Strong stakeholder management skills across business, technology, architecture, and executive audiences.
- Proven ability to influence senior leadership and drive strategic technology decisions.
- Databricks Certified Data Engineer Professional.
- Databricks Certified Lakehouse Platform Architect.
- Experience with MLOps and AI Engineering platforms.
- Experience leading enterprise migration programmes from legacy data platforms.
- Financial Services, Data Analytics, Benchmarking, or Information Services industry experience.
- Experience working in scaled Agile environments and PI Planning.
'/%3e%3cpath%20d='M101.744%2012.4726L102.894%2016.1947L99.0021%2013.2201L101.744%2012.4726ZM105.3%208.40937H100.488L99.0026%203.59473L97.5116%208.41047L92.7%208.40385L96.5965%2011.3834L95.1055%2016.1942L99.0021%2013.2196L101.408%2011.3834L105.3%208.40992V8.40937Z'%20fill='white'/%3e%3cdefs%3e%3clinearGradient%20id='paint0_linear_5593_1381'%20x1='90'%20y1='9.44471'%20x2='99'%20y2='9.44471'%20gradientUnits='userSpaceOnUse'%3e%3cstop%20stop-color='%2300AD70'/%3e%3cstop%20offset='0.998594'%20stop-color='%2300B67A'/%3e%3cstop%20offset='1'%20stop-color='%23DCDCE6'/%3e%3c/linearGradient%3e%3c/defs%3e%3c/svg%3e){kind=small}
