Databricks Architect

Tata Consultancy Services; Irvine (CA); On-site; USD 170,000 - 210,000; Full time

Job summary

Tata Consultancy Services Limited in California seeks a seasoned Databricks Architect to design and implement enterprise Databricks Lakehouse platforms across cloud environments. The role emphasizes strong data governance, security, and leadership to guide multi-disciplinary teams.

The successful candidate will bring 12+ years of IT experience, 5+ years in Databricks architecture, hands-on Spark/PySpark, and deep knowledge of lakehouse patterns, Unity Catalog, and data modeling.

Qualifications

• 12+ years of IT experience.
• 5+ years of Databricks architecture experience.
• Strong lakehouse architecture expertise.
• Hands-on Spark/PySpark experience.
• Data governance and security expertise.
• Stakeholder management and leadership experience.

Responsibilities

• Design and implement enterprise-scale Databricks Lakehouse platforms.
• Define end-to-end data architecture for batch, streaming, and real-time analytics.
• Develop cloud-native data platform strategies aligned with business objectives.
• Establish architectural standards, governance frameworks and best practices.
• Lead modernization initiatives including legacy data warehouse migration to Databricks.
• Design scalable ETL/ELT pipelines and medallion architecture across platforms.

Skills

Lakehouse architecture

Spark/PySpark

Data governance

Stakeholder management

Leadership

Databricks platform

Cloud architecture

Tools

Databricks Lakehouse Platform

Photon Engine

Unity Catalog

Databricks Workflows

MLflow

Auto Loader

Structured Streaming

Databricks SQL

Databricks Asset Bundles

PySpark

Spark SQL

Scala

Python

SQL

Terraform

GitHub

Docker

Job description

Former Databricks Employee (Non-Negotiable Requirement)

12+ Years of IT Experience

5+ Years of Databricks Architecture Experience

Strong Lakehouse Architecture Expertise

Hands-on Spark/PySpark Experience

Data Governance & Security Expertise

Stakeholder Management and Leadership Experience

Preferred

BFSI Domain Experience

Databricks Technologies

• Databricks Lakehouse Platform


• Photon Engine


• Unity Catalog


• Databricks Workflows


• MLflow


• Auto Loader


• Structured Streaming


• Databricks SQL


• Databricks Asset Bundles


• PySpark


• Spark SQL


• Scala (preferred)


• Python


• SQL Optimization


• Data Modeling


• Data Warehousing


• AWS Databricks Ecosystem

DevOps & Automation

• Terraform


• GitHub


• Docker

Data Governance & Security

• Unity Catalog


• Data Quality Frameworks


• Data Classification & Governance

Domain Experi ence (Preferred)

• Capital Markets


• Wealth Management


• Insurance


• Lending & Consumer Finance


• Regulatory Reporting and Compliance

Certifications (Preferred)

• Databricks Certified Solution Architect


• Azure Solutions Architect Expert

Architecture & Strategy

• Design and implement enterprise-scale Databricks Lakehouse platforms.


• Define end-to-end data architecture for batch, streaming, and real-time analytics solutions.


• Develop cloud-native data platform strategies aligned with business objectives.


• Establish architectural standards, governance frameworks, and best practices.


• Lead modernization initiatives involving legacy data warehouse migration to Databricks.


• Design scalable ETL/ELT pipelines using Databricks, Apache Spark, and Delta Lake.


• Build medallion architecture (Bronze, Silver, Gold layers) for enterprise data platforms.


• Integrate data from multiple sources including databases, APIs, SaaS platforms, and streaming systems.


• Implement data quality, lineage, observability, and monitoring frameworks.


• Optimize data ingestion and transformation processes for performance and scalability.


• Architect Databricks solutions on Azure, AWS, or Google Cloud Platform.


• Design secure networking, storage, identity management, and access controls.


• Implement Infrastructure as Code (Terraform, ARM, CloudFormation).


• Optimize platform usage and cloud costs through governance and monitoring.


• Lead Databricks workspace design and multi-environment deployment strategies.

AI, ML & Advanced Analytics

• Architect Databricks solutions supporting Machine Learning and Generative AI workloads.


• Enable enterprise AI capabilities leveraging Databricks Mosaic AI and GenAI frameworks.


• Collaborate with Data Scientists and AI teams to operationali ze ML models.


• Establish MLOps best practices and governance standards.

Security & Governance

• Implement Unity Catalog for enterprise-wide governance.


• Define data security controls including RBAC, ABAC, encryption, and access management.


• Ensure compliance with industry standards including SOX, PCI-DSS, HIPAA, GDPR, and BFSI regulatory requirements.


• Establish data governance, metadata management, and audit frameworks.

Leadership & Stakeholder Management

• Act as the primary Databricks subject matter expert.


• Engage with executive stakeholders to define data transformation roadmaps.


• Conduct architecture reviews and provide technical leadership.


• Mentor data engineers, platform engineers, and architects.


• Collaborate with business, infrastructure, security, and application teams.