Databricks Lead

Tata Consultancy Services; Minneapolis (MN); On-site; USD 130,000 - 150,000; Full time

Benefits offered by this job

Discretionary Annual Incentive

Comprehensive Medical Coverage: Health

Family Support: Parental Leaves

Commuter Benefits & Training Reimburse

Vacation, Time Off & Holidays

401K Plan & Performance Bonus

Job summary

Tata Consultancy Services Limited in Minneapolis, MN is seeking an experienced Lead Data Engineer to drive the enterprise data platform strategy using Databricks and cloud-native technologies. You will define architectures, lead design reviews, and deliver scalable data pipelines across batch and streaming workloads.

You will mentor data engineering teams, establish governance, and collaborate with data science and AI teams to operationalize ML workloads, ensuring reliability, security, and

Qualifications

• 12+ years of experience in Data Engineering and Analytics.
• 5+ years of hands-on Databricks experience.
• 3+ years leading enterprise-scale data engineering teams.
• Proven experience building large-scale cloud data platforms.
• Experience leading distributed teams in Agile delivery environments

Responsibilities

• Define and execute the enterprise data platform strategy leveraging Databricks and cloud-native technologies.
• Establish reference architectures, standards, and engineering best practices.
• Lead technical design reviews and architecture governance forums.
• Create reusable frameworks and accelerators to improve productivity and quality.
• Design and oversee implementation of scalable batch, streaming, and real-time data pipelines.
• Lead development of data products supporting analytics, reporting, data science, and AI initiatives.
• Ensure high standards for reliability, scalability, observability, security, and maintainability.
• Drive adoption of DataOps, DevOps, and CI/CD practices across engineering teams.
• Define enterprise Lakehouse architecture patterns.
• Establish data models, storage strategies, and data lifecycle management processes.
• Lead implementation of data governance, metadata management, lineage, and access controls.
• Ensure platform compliance with organizational security and regulatory requirements.
• Partner with data science and AI teams to operationalize machine learning workloads.
• Enable model training, deployment, monitoring, and lifecycle management.
• Support GenAI, predictive analytics, and intelligent automation initiatives.
• Build scalable data foundations for AI-driven business capabilities.
• Lead and mentor data engineering teams.
• Manage technical delivery across multiple workstreams and vendors.
• Collaborate with business stakeholders to translate requirements into scalable solutions.
• Provide regular updates to leadership on delivery progress, risks, and architectural decisions.

Skills

Python

Apache Spark

Education

Bachelor of Computer Science

Tools

Databricks Lakehouse Platform

Delta Lake

Unity Catalog

Databricks Workflows

Databricks SQL

Delta Live Tables

MLflow

Apache Spark

PySpark

Data Modeling

ETL/ELT Architecture

Streaming Data Processing

Data Quality Frameworks

Event-Driven Architectures

Programming

Python

SQL

Job description

Job Description

Must Have Technical/Functional Skills

• * Databricks Lakehouse Platform
• * Delta Lake
• * Unity Catalog
• * Databricks Workflows
• * Databricks SQL
• * Delta Live Tables
• * MLflow
• * Apache Spark
• * PySpark
• * Data Modeling
• * ETL/ELT Architecture
• * Streaming Data Processing
• * Data Quality Frameworks
• * Event-Driven Architectures
• * Programming
• * Python
• * SQL

Roles & Responsibilities

• * Define and execute the enterprise data platform strategy leveraging Databricks and cloud-native
• technologies.
• * Establish reference architectures, standards, and engineering best practices.
• * Lead technical design reviews and architecture governance forums.
• * Create reusable frameworks and accelerators to improve productivity and quality.
• * Design and oversee implementation of scalable batch, streaming, and real-time data pipelines.
• * Lead development of data products supporting analytics, reporting, data science, and AI
• initiatives.
• * Ensure high standards for reliability, scalability, observability, security, and maintainability.
• * Drive adoption of DataOps, DevOps, and CI/CD practices across engineering teams.
• * Define enterprise Lakehouse architecture patterns.
• * Establish data models, storage strategies, and data lifecycle management processes.
• * Lead implementation of data governance, metadata management, lineage, and access controls.
• * Ensure platform compliance with organizational security and regulatory requirements.
• * Partner with data science and AI teams to operationalize machine learning workloads.
• * Enable model training, deployment, monitoring, and lifecycle management.
• * Support GenAI, predictive analytics, and intelligent automation initiatives.
• * Build scalable data foundations for AI-driven business capabilities.
• * Lead and mentor data engineering teams.
• * Manage technical delivery across multiple workstreams and vendors.
• * Collaborate with business stakeholders to translate requirements into scalable solutions.
• * Provide regular updates to leadership on delivery progress, risks, and architectural decisions.

Generic Managerial Skills

• * 12+ years of experience in Data Engineering and Analytics.
• * 5+ years of hands-on Databricks experience.
• * 3+ years leading enterprise-scale data engineering teams.
• * Proven experience building large-scale cloud data platforms.
• * Experience leading distributed teams in Agile delivery environments

Salary Range: $130,000-$150,000 a year

TCS Employee Benefits Summary:

• Discretionary Annual Incentive.
• Comprehensive Medical Coverage: Medical & Health, Dental & Vision, Disability Planning & Insurance, Pet Insurance Plans.
• Family Support: Maternal & Parental Leaves.
• Insurance Options: Auto & Home Insurance, Identity Theft Protection.
• Convenience & Professional Growth: Commuter Benefits & Certification & amp; Training Reimbursement.
• Time Off: Vacation, Time Off, Sick Leave & Holidays.
• Legal & Financial Assistance: Legal Assistance, 401K Plan, Perform ance Bonus, College Fund, Student Loan Refinancing.

Location

Minneapolis, MN

Function

TECHNOLOGY

Role

Lead

ID

436825

Skills

Python | Apache Spark

Salary Range

$130,000 - $150,000

Desired Candidate Profile

Qualifications

BACHELOR OF COMPUTER SCIENCE