Senior Databricks Lakehouse Architect for AI Analytics

Tata Consultancy Services; Marlborough (MA); On-site; USD 110,000 - 140,000; Full time

Benefits offered by this job

Discretionary Annual Incentive

Comprehensive Medical Coverage

401K Plan

Certification & Training Reimbursement

Job summary

Tata Consultancy Services in Marlborough, MA seeks a senior data engineer with 10+ years of data engineering experience and 3+ years focused on Databricks/Spark. You will design scalable data pipelines, implement Delta Lake and Lakeflow, and govern data with Unity Catalog.

Proficiency in Python, PySpark, and SQL is essential, plus cloud experience and CI/CD practices. You will collaborate across teams to build Gold-layer datasets, ensure data quality, and optimize performance and costs in a

Qualifications

• 10+ years hands-on data engineering experience with focus on Databricks/Spark.
• Deep Databricks Lakehouse Platform expertise including Delta Lake.
• Structured streaming, Lakeflow Declarative Pipelines, Databricks SQL, and cluster/serverless tuning.
• Unity Catalog governance with metric views, domains, glossary for governed KPIs.
• Experience with AI/BI Genie, Genie Spaces and Mosaic AI for AI/agent workloads.
• Expert Python/PySpark, advanced SQL skills.
• Data warehousing concepts (Kimball) and ETL/ELT design patterns.
• Cloud provider experience (Azure/AWS/GCP) and S3-related services.
• Software engineering mindset: Git, code reviews, testing, CI/CD.

Responsibilities

• Data Pipeline Development: design, code, and deploy scalable batch/streaming pipelines with PySpark and Lakeflow.
• Ingest data from POS, e-commerce, loyalty, and marketing clouds.
• Data Modeling & Transformation in Medallion architecture (Bronze/Silver/Gold).
• Define governed KPIs on Gold layer using Unity Catalog metric views.
• Implement data quality and cleansing for Customer 360 data.
• Tune Databricks jobs and Spark clusters for performance and cost-efficiency.
• Manage IaC/CI-CD with Terraform and GitHub Actions; collaborate with DevOps.
• Apply data governance features: lineage, fine-grained access, masking; ensure security.
• collaborate with analysts to deliver consumption-ready datasets.
• Govern Unity Catalog semantic modeling and Genie-enabled analytics delivery.

Skills

Python

PySpark

SQL

Cloud experience

Data Modeling

CI/CD practices

Communication

Tools

Databricks Lakehouse Platform

Delta Lake

Lakeflow Declarative Pipelines

Unity Catalog

Genie Spaces

Databricks One

Terraform

Job description

Tata Consultancy Services in Marlborough, MA seeks a senior data engineer with 10+ years of data engineering experience and 3+ years focused on Databricks/Spark. You will design scalable data pipelines, implement Delta Lake and Lakeflow, and govern data with Unity Catalog.

Proficiency in Python, PySpark, and SQL is essential, plus cloud experience and CI/CD practices. You will collaborate across teams to build Gold-layer datasets, ensure data quality, and optimize performance and costs in a