Databricks Solution Architect

CRIF India; Pune District; On-site; INR 4,000,000 - 7,000,000; Full time

Job summary

CRIF India is seeking a senior Architect to own the end-to-end Databricks Lakehouse on AWS, including landing zone, governance, and integration with the operational data layer. You will be the design authority and escalation point for platform decisions, with focus on security, cost, and multi-country compliance.

Lead architecture reviews, mentor engineering teams across data, ML, and BI streams, and collaborate with global stakeholders to drive standardization and scalable delivery.

Qualifications

• Deep Databricks architecture expertise including Unity Catalog and lakehouse governance.
• Strong AWS architecture experience with VPCs, IAM, S3 and cross-account access.
• Extensive practice in large-scale, low-latency, multi-tenant data platforms.
• Experience implementing data governance and regulatory compliance in BFSI contexts.

Responsibilities

• Define and evolve the target Databricks Lakehouse architecture on AWS.
• Own the Databricks landing zone: workspaces, account structure, networking, encryption.
• Design Unity Catalog governance: metastore, locations, credentials, access, lineage.
• Architect integration between DynamoDB and Lakehouse with streaming and replay semantics.
• Set standards for data modelling, table formats (Delta/Iceberg), CI/CD and quality.
• Own platform cost architecture: DBU modelling, commitments, compute policies, chargeback.
• Define security posture for multi-country BFSI footprint: data residency, retention, audit readiness.
• Lead architecture reviews and defend design choices to stakeholders and boards.
• Mentor engineering, ML and BI streams; unblock complex technical problems.

Skills

Databricks architecture

AWS architecture

Large-scale distributed data

Spark internals

Security & regulatory compliance

Tools

Terraform

Databricks Platform

AWS Lake Formation

Job description

Own the end-to-end architecture of the Databricks Lakehouse platform on AWS — the landing zone, governance model, integration with the operational data layer, and the standards every engineering stream builds against. You will be the design authority and the point of technical escalation for the platform.

Key Responsibilities

• Define and evolve the target Lakehouse architecture: medallion layers, catalog structure, storage strategy, compute topology.
• Own the Databricks landing zone on AWS — workspaces, account structure, networking (PrivateLink, NCC, VPC design), encryption and key management.
• Design the Unity Catalog governance model: metastore layout, external locations, storage credentials, access control, lineage and audit.
• Architect integration between the operational store (DynamoDB) and the analytical Lakehouse, including streaming, replay and consistency semantics.
• Set standards for data modelling, table formats (Delta/Iceberg), naming, CI/CD, environment promotion and code quality.
• Own platform cost architecture — DBU consumption modelling, commitment planning, compute policies, chargeback and optimisation.
• Define security and compliance posture for a multi-country BFSI footprint: data residency, PII handling, retention, regulatory audit readiness.
• Lead architecture reviews, produce decision records and defend design choices to senior stakeholders and global review boards.
• Mentor engineering, ML and BI streams; unblock complex technical problems.

Must-Have Skills

• Deep Databricks architecture expertise — Unity Catalog, Delta Lake, workspace and account administration, serverless and classic compute.
• Strong AWS architecture: VPC and hybrid networking, PrivateLink, IAM and cross-account trust, S3, Glue, Lake Formation, KMS.
• Large-scale distributed data architecture and Spark internals.
• Proven experience designing platforms handling high-volume, low-latency and multi-tenant workloads.
• Data governance, security and regulatory compliance in a financial-services context.

Good to Have

• Databricks Certified Data Engineer Professional and/or Databricks Platform Architect accreditation.
• AWS Solutions Architect Professional.
• Multi-cloud experience — Azure Databricks, cross-cloud commitment models.
• Terraform / infrastructure-as-code for Databricks and AWS.

Additional Info

• Comfortable working in a regulated, audit-heavy environment with strict change control.
• Collaborative working style across engineering, product, risk and country teams.
• Willingness to work with distributed/global stakeholders across time zones.