DataBricks Specialist

Tata Consultancy Services; New York (NY); On-site; USD 130,000 - 150,000; Full time

Benefits offered by this job

Discretionary incentive

Medical coverage

Dental & Vision

Parental leave

401K plan

Certification reimbursement

Commuter benefits

Vacation & holidays

Job summary

Tata Consultancy Services in United States is seeking an experienced data engineer/architect to design end-to-end data pipelines, lakehouse patterns, and ETL/ELT processes using Apache Spark, Delta Lake, and Python or SQL.

You will optimize performance, manage ML workflows with MLflow and Unity Catalog, and collaborate with data scientists and engineers to translate business needs into scalable, technical solutions.

Qualifications

• 5 to 10+ years in data engineering, software development, or solution architecture.
• Hands-on Databricks production experience.
• Strong proficiency in Python, SQL, or Scala.
• Deep knowledge of Apache Spark runtime internals and Delta Lake.
• Working knowledge of cloud platforms (Azure, AWS, or GCP).
• Databricks Certified Data Engineer or Solutions Architect credentials preferred.

Responsibilities

• Design end-to-end data pipelines, ETL/ELT processes, and lakehouse storage patterns using Spark, Delta Lake, and Python or SQL.
• Optimize cluster sizing, query execution, shuffles, partitions, and storage layouts for large-scale data workflows.
• Implement MLOps workflows, manage model lifecycles, or deploy generative AI and vector search solutions using MLflow and Unity Catalog.
• Partner with enterprise stakeholders, data scientists, and software engineers to translate business requirements into technical solutions.
• Oversee security configurations, access controls, workspace administration, and CI/CD deployment pipelines.

Skills

Databricks production experience

Python

SQL

Scala

Apache Spark

Delta Lake

Cloud platforms (Azure/AWS/GCP)

MLflow

Unity Catalog

Education

Bachelor of Computer Science

Tools

Delta Lake

MLflow

Unity Catalog

Job description

Job Description

Must Have Technical/Functional Skills

• Experience: 5 to 10+ years in data engineering, software development, or solution architecture, with hands-on Databricks production experience.
• Technical Stack: Strong proficiency in Python, SQL, or Scala; deep knowledge of Apache Spark runtime internals and Delta Lake.
• Cloud Ecosystems: Working knowledge of major cloud platforms like Microsoft Azure, AWS, or GCP.
• Certifications: Preferred credentials include Databricks Certified Data Engineer or Solutions Architect credentials. []

Roles & Responsibilities

• Architecture & Development: Design end-to-end data pipelines, ETL/ELT processes, and lakehouse storage patterns using Apache Spark, Delta Lake, and Python or SQL.
• Performance Tuning: Optimize cluster sizing, query execution, shuffles, partitions, and storage layouts for large-scale data workflows. [1]
• AI & Machine Learning Integration: Implement MLOps workflows, manage model lifecycles, or deploy generative AI and vector search solutions using tools like MLflow and Unity Catalog. [1]
• Collaboration & Consulting: Partner with enterprise stakeholders, data scientists, and software engineers to translate business requirements into technical solutions. [1]
• Administration & Governance: Oversee security configurations, access controls, workspace administration, and CI/CD deployment pipelines.

TCS Employee Benefits Summary

• Discretionary Annual Incentive.
• Comprehensive Medical Coverage: Medical & Health, Dental & Vision, Disability Planning & Insurance, Pet Insurance Plans.
• Family Support: Maternal & Parental Leaves.
• Insurance Options: Auto & Home Insurance, Identity Theft Protection.
• Convenience & Professional Growth: Commuter Benefits & Certification & Training Reimbursement.
• Time Off: Vacation, Time Off, Sick Leave & Holidays.
• Legal & Financial Assistance: Legal Assistance, 401K Plan, Performance Bonus, College Fund, Student Loan Refinancing.
• Salary Range: $130,000 - $150,000 a year

Qualifications:

BACHELOR OF COMPUTER SCIENCE