Databricks Tech Lead

Tata Consultancy Services Limited; Pleasanton (CA); On-site; USD 130,000 - 140,000; Full time

Benefits offered by this job

Discretionary Incentive

Medical Coverage

Parental Leave

Identity Theft Protection

Training Reimbursement

Vacation & Holidays

401K Plan

Job summary

Tata Consultancy Services Limited in Pleasanton, CA is seeking a Technical Lead to guide data modernization programs. You will assess legacy data platforms, define migration roadmaps to Databricks Lakehouse, and lead cross-functional teams through POC, pilot, and large-scale migrations.

The role requires 10+ years in data engineering and hands-on experience with Databricks, Delta Lake, PySpark, and Azure data services, plus strong governance, CI/CD, and security practices.

Qualifications

• 10+ years of data engineering or data architecture experience.
• 6+ years with Databricks in large-scale environments.
• Expertise in Databricks, Spark, PySpark, SQL, and Delta Lake.
• Experience migrating legacy platforms to Databricks Lakehouse.

Responsibilities

• Lead current-state assessment of data platforms and pipelines.
• Define migration strategy to a Databricks Lakehouse architecture.
• Design target-state data architecture using cloud-native services.
• Lead migration of ETL/analytics workloads to Databricks with PySpark/SQL.
• Mentor teams on Databricks architecture and coding standards.
• Establish CI/CD and governance for data solutions.

Skills

Data engineering

Data architecture

ETL modernization

Enterprise data platform migration

Technical leadership

Migration strategy

Discovery workshops

Architecture reviews

Education

Bachelor of Computer Science

Tools

Databricks

Apache Spark

PySpark

SQL

Delta Lake

Databricks Workflows/Jobs

Alteryx

Azure Databricks

ADLS

Azure Data Factory

Purview

Key Vault

Azure DevOps

Job description

Job Description

Must Have Technical/Functional Skills

• 10+ years of experience in Data Engineering, Data Architecture, ETL Modernization, or Enterprise Data Platform Migration.
• 6+ years of hands‑on Databricks experience in large-scale enterprise environments with Data Modernization or migration to Databricks expertise.
• Strong expertise in Databricks, Apache Spark, PySpark, SQL, Delta Lake, and Databricks Workflows/Jobs.
• Proven experience assessing and migrating legacy data platforms, ETL solutions, and data warehouses to Databricks.
• Experience with Alteryx workflow assessment, migration planning, and modernization.
• Strong understanding of data modeling, data pipelines, source-to-target mappings, data quality, and performance optimization.
• Experience designing scalable Lakehouse architectures, including Medallion Architecture and Delta Lake.
• Knowledge of cloud data architecture, security, governance, monitoring, CI/CD, and DevOps practices.
• Hands‑on experience with Azure Databricks, ADLS, Azure Data Factory, Purview, Key Vault, and Azure DevOps.
• Ability to lead technical discovery workshops, define migration strategies, prepare architecture documentation, and provide technical leadership throughout delivery.

Roles & Responsibilities

• Lead the assessment of current-state data platforms, including source systems, data warehouses, databases, ETL/ELT pipelines, reporting environments, and Alteryx workflows.
• Define and drive the migration and modernization strategy for legacy data platforms to Databricks Lakehouse Architecture.
• Analyze existing data assets, transformation logic, dependencies, source-to-target mappings, data quality requirements, and performance considerations.
• Identify workloads suitable for lift‑and‑shift migration versus those requiring redesign, refactoring, consolidation, or retirement.
• Design target‑state data architecture leveraging Databricks, Delta Lake, Medallion Architecture, and cloud‑native data services.
• Lead the migration of legacy ETL, data integration, and analytics workloads to Databricks using PySpark, SQL, Databricks Workflows, and related technologies.
• Assess and modernize Alteryx workflows and legacy data processing solutions into scalable Databricks‑based implementations.
• Define best practices for data ingestion, transformation, orchestration, security, governance, monitoring, metadata management, and lineage tracking.
• Conduct technical discovery workshops and architecture reviews with business, data engineering, analytics, and platform stakeholders.
• Create architecture documents, migration roadmaps, dependency assessments, implementation plans, effort estimates, and risk mitigation strategies.
• Provide hands‑on technical leadership during proof‑of‑concept, pilot, and large‑scale migration initiatives.
• Guide and mentor development teams on Databricks architecture, coding standards, performance tuning, and engineering best practices.
• Collaborate with cross‑functional teams to ensure migration and modernization initiatives align with business objectives and delivery priorities.
• Establish CI/CD, DevOps, testing, and deployment standards for Databricks‑based data solutions.
• Ensure compliance with enterprise security, governance, data privacy, and operational standards throughout the migration lifecycle.
• Act as the primary technical lead for architecture decisions, solution reviews, and stakeholder communications related to Databricks modernization programs.

Generic Managerial Skills, If any

• Strong stakeholder management and communication skills with the ability to lead technical discussions and influence architecture decisions.
• Proven experience leading cross‑functional teams in enterprise data migration and modernization programs.
• Strong planning, problem‑solving, and delivery management skills to drive complex initiatives from assessment through implementation.

Salary Range: $130,000-$140,000 a year

TCS Employee Benefits Summary:

• Discretionary Annual Incentive.
• Comprehensive Medical Coverage: Medical & Health, Dental & Vision, Disability Planning & Insurance, Pet Insurance Plans.
• Family Support: Maternal & Parental Leaves.
• Insurance Options: Auto & Home Insurance, Identity Theft Protection.
• Convenience & Professional Growth: Commuter Benefits & Certification & Training Reimbursement.
• Time Off: Vacation, Time Off, Sick Leave & Holidays.
• Legal & Financial Assistance: Legal Assistance, 401K Plan, Performance Bonus, College Fund, Student Loan Refinancing.

Location

Pleasanton, CA

Function

TECHNOLOGY

Role

Technical Lead

ID

436399

Skills

SQL | Apache Spark

Salary Range

$130,000 - $140,000

Desired Candidate Profile

Qualifications

BACHELOR OF COMPUTER SCIENCE