Databricks Architect

Tata Consultancy Services Limited; Irvine (CA); Presencial; USD 120.000 - 150.000; Jornada completa

Descripción de la vacante

Tata Consultancy Services is seeking an experienced Databricks Architect in Irvine, CA. You will design and implement enterprise Databricks Lakehouse platforms, define data architectures, and lead modernization efforts across cloud environments.

The role requires deep expertise in Spark, Lakehouse, governance, and stakeholder leadership to deliver scalable data platforms for complex business needs.

Formación

• 12+ years of IT experience required.
• 5+ years of Databricks architecture experience.
• Strong lakehouse architecture expertise.
• Hands-on Spark/PySpark experience.
• Experience with enterprise cloud data platforms, data governance and security.
• Proven stakeholder management and leadership skills.

Responsabilidades

• Design and implement enterprise-scale Databricks Lakehouse platforms.
• Define end-to-end data architecture for batch, streaming, and real-time analytics.
• Develop cloud-native data platform strategies aligned with business goals.
• Establish architectural standards, governance, and best practices.
• Lead modernization of legacy data warehouse migrations to Databricks.
• Design scalable ETL/ELT pipelines using Spark and Delta Lake.
• Architect data integration from multiple sources and ensure data quality.
• Lead Databricks workspace design and multi-environment deployment strategies.
• Collaborate with data scientists to operationalize ML models.
• Ensure security and governance with Unity Catalog and RBAC.

Conocimientos

Databricks Architecture

Spark/PySpark

Lakehouse Architecture

Stakeholder Management

Leadership

Cloud Platform Experience

Herramientas

Databricks Lakehouse Platform

Delta Lake

Photon Engine

Unity Catalog

Delta Live Tables

Databricks Workflows

MLflow

Auto Loader

Structured Streaming

Databricks SQL

Databricks Asset Bundles

Descripción del empleo

Job Description

Databricks Architect

Must Have Technical/Functional Skills

Mandatory

Former Databricks Employee (Non-Negotiable Requirement)

12+ Years of IT Experience

5+ Years of Databricks Architecture Experience

Strong Lakehouse Architecture Expertise

Hands-on Spark/PySpark Experience

Enterprise Cloud Data Platform Design Experience

Data Governance & Security Expertise

Stakeholder Management and Leadership Experience

Preferred

BFSI Domain Experience

GenAI / AI Platform Architecture Experience

Large-scale Cloud Migration Experience

Pre-sales / Solution Consulting Experience

Databricks Technologies

• Databricks Lakehouse Platform
• Delta Lake
• Photon Engine
• Unity Catalog
• Delta Live Tables (DLT)
• Databricks Workflows
• MLflow
• Auto Loader
• Structured Streaming
• Databricks SQL
• Databricks Asset Bundles

Data Engineering

• Apache Spark
• PySpark
• Spark SQL
• Scala (preferred)
• Python
• SQL Optimization
• Data Modeling
• Data Warehousing

Cloud Platforms

• Microsoft Azure (Preferred)
• Azure Data Factory
• Azure Synapse Analytics
• Azure Storage
• Azure Key Vault
• Azure Entra ID (Azure AD)

OR

• AWS Databricks Ecosystem
• Google Cloud Databricks Ecosystem

DevOps & Automation

• Terraform
• GitHub
• Azure DevOps
• CI/CD Pipelines
• Docker
• Kubernetes

Data Governance & Security

• Unity Catalog
• Apache Ranger
• Data Lineage Tools
• Data Quality Frameworks
• Data Classification & Governance

________________________________________

Domain Experi Ence (Preferred)

• Banking & Financial Services (BFSI)
• Capital Markets
• Wealth Management
• Insurance
• Lending & Consumer Finance
• Regulatory Reporting and Compliance

________________________________________

Certifications (Preferred)

• Databricks Certified Data Engineer Professional
• Databricks Certified Solution Architect
• Databricks Certified Machine Learning Professional
• Azure Solutions Architect Expert
• Azure Data Engineer Associate

Roles & Responsibilities

Architecture & Strategy

• Design and implement enterprise-scale Databricks Lakehouse platforms.
• Define end-to-end data architecture for batch, streaming, and real-time analytics solutions.
• Develop cloud-native data platform strategies aligned with business objectives.
• Establish architectural standards, governance frameworks, and best practices.
• Lead modernization initiatives involving legacy data warehouse migration to Databricks.

Data Engineering & Integration

• Design scalable ETL/ELT pipelines using Databricks, Apache Spark, and Delta Lake.
• Build medallion architecture (Bronze, Silver, Gold layers) for enterprise data platforms.
• Integrate data from multiple sources including databases, APIs, SaaS platforms, and streaming systems.
• Implement data quality, lineage, observability, and monitoring frameworks.
• Optimize data ingestion and transformation processes for performance and scalability.

Cloud & Platform Engineering

• Architect Databricks solutions on Azure, AWS, or Google Cloud Platform.
• Design secure networking, storage, identity management, and access controls.
• Implement Infrastructure as Code (Terraform, ARM, CloudFormation).
• Optimize platform usage and cloud costs through governance and monitoring.
• Lead Databricks workspace design and multi-environment deployment strategies.

AI, ML & Advanced Analytics

• Architect Databricks solutions supporting Machine Learning and Generative AI workloads.
• Design ML lifecycle management using MLflow.
• Enable enterprise AI capabilities leveraging Databricks Mosaic AI and GenAI frameworks.
• Collaborate with Data Scientists and AI teams to operationali ze ML models.
• Establish MLOps best practices and governance standards.

Security & Governance

• Implement Unity Catalog for enterprise-wide governance.
• Define data security controls including RBAC, ABAC, encryption, and access management.
• Ensure compliance with industry standards including SOX, PCI-DSS, HIPAA, GDPR, and BFSI regulatory requirements.
• Establish data governance, metadata management, and audit frameworks.

Leadership & Stakeholder Management

• Act as the primary Databricks subject matter expert.
• Engage with executive stakeholders to define data transformation roadmaps.
• Conduct architecture reviews and provide technical leadership.
• Mentor data engineers, platform engineers, and architects.
• Collaborate with business, infrastructure, security, and application teams.

Salary Range- $120,000-$150,000 a year