Databricks Lakehouse Lead - Data Platform & AI

Tata Consultancy Services Limited; Minneapolis (MN); On-site; USD 130,000 - 150,000; Full time

Benefits offered by this job

Discretionary Annual Incentive

Comprehensive Medical Coverage: Health

Family Support: Parental Leaves

Commuter Benefits & Training Reimburse

Vacation, Time Off & Holidays

401K Plan & Performance Bonus

Job summary

Tata Consultancy Services Limited in Minneapolis, MN is seeking an experienced Lead Data Engineer to drive the enterprise data platform strategy using Databricks and cloud-native technologies. You will define architectures, lead design reviews, and deliver scalable data pipelines across batch and streaming workloads.

You will mentor data engineering teams, establish governance, and collaborate with data science and AI teams to operationalize ML workloads, ensuring reliability, security, and

Qualifications

• 12+ years of experience in Data Engineering and Analytics.
• 5+ years of hands-on Databricks experience.
• 3+ years leading enterprise-scale data engineering teams.
• Proven experience building large-scale cloud data platforms.
• Experience leading distributed teams in Agile delivery environments

Responsibilities

• Define and execute the enterprise data platform strategy leveraging Databricks and cloud-native technologies.
• Establish reference architectures, standards, and engineering best practices.
• Lead technical design reviews and architecture governance forums.
• Create reusable frameworks and accelerators to improve productivity and quality.
• Design and oversee implementation of scalable batch, streaming, and real-time data pipelines.
• Lead development of data products supporting analytics, reporting, data science, and AI initiatives.
• Ensure high standards for reliability, scalability, observability, security, and maintainability.
• Drive adoption of DataOps, DevOps, and CI/CD practices across engineering teams.
• Define enterprise Lakehouse architecture patterns.
• Establish data models, storage strategies, and data lifecycle management processes.
• Lead implementation of data governance, metadata management, lineage, and access controls.
• Ensure platform compliance with organizational security and regulatory requirements.
• Partner with data science and AI teams to operationalize machine learning workloads.
• Enable model training, deployment, monitoring, and lifecycle management.
• Support GenAI, predictive analytics, and intelligent automation initiatives.
• Build scalable data foundations for AI-driven business capabilities.
• Lead and mentor data engineering teams.
• Manage technical delivery across multiple workstreams and vendors.
• Collaborate with business stakeholders to translate requirements into scalable solutions.
• Provide regular updates to leadership on delivery progress, risks, and architectural decisions.

Skills

Python

Apache Spark

Education

Bachelor of Computer Science

Tools

Databricks Lakehouse Platform

Delta Lake

Unity Catalog

Databricks Workflows

Databricks SQL

Delta Live Tables

MLflow

Apache Spark

PySpark

Data Modeling

ETL/ELT Architecture

Streaming Data Processing

Data Quality Frameworks

Event-Driven Architectures

Programming

Python

SQL

Job description

Tata Consultancy Services Limited in Minneapolis, MN is seeking an experienced Lead Data Engineer to drive the enterprise data platform strategy using Databricks and cloud-native technologies. You will define architectures, lead design reviews, and deliver scalable data pipelines across batch and streaming workloads.

You will mentor data engineering teams, establish governance, and collaborate with data science and AI teams to operationalize ML workloads, ensuring reliability, security, and