Senior Databricks Data Engineer - Spark, Lakehouse, AWS

Tata Consultancy Services Limited; Boynton Beach (FL); On-site; USD 100,000 - 130,000; Full time

Benefits offered by this job

Discretionary Annual Incentive

Commuter Benefits

Certification & Training Reimbursement

Healthcare: Medical Coverage

Dental & Vision Coverage

Disability Planning & Insurance

401K Plan

Family Support: Leaves

Job summary

Tata Consultancy Services Limited in Boynton Beach, FL seeks a Senior Data Engineer to lead design and implementation of large-scale data pipelines. You will work with Databricks and PySpark to build scalable batch and real-time data solutions and mentor junior engineers.

Role requires 8+ years in data engineering, Delta Lake mastery, and strong SQL/PLSQL skills. On-site role with a focus on cost optimization and reliable data platforms.

Qualifications

• 8+ years in Data Engineering, ETL/ELT and Data Warehousing.
• 5+ years with Databricks, PySpark and Apache Spark.
• Strong Delta Lake, Lakehouse architecture and large-scale data processing expertise.
• Advanced SQL and PL/SQL development skills.
• Experience designing scalable batch and real-time data pipelines.
• Knowledge of dimensional/relational data modeling.
• Experience with Spark SQL, DataFrames, Hive and distributed computing.
• Hands-on AWS services experience (S3, IAM, Glue, Lambda, Redshift).
• CI/CD, Git, DevOps and deployment automation knowledge.
• Excellent analytical and stakeholder management skills; prior leadership experience.

Responsibilities

• Gather and analyze business/technical requirements from stakeholders.
• Design end-to-end data solutions, architectures, and scalable data models.
• Develop and optimize ingestion, transformation, and analytics pipelines with Databricks and PySpark.
• Create and maintain conceptual, logical, and physical data models.
• Lead design activities and document High-Level/Low-Level designs.
• Implement data quality, governance, and performance optimization practices.
• Collaborate with Product Owners, Architects, and BAs to define scope and plans.
• Provide technical leadership and mentoring to onshore/offshore teams.
• Conduct design/code reviews and troubleshoot to ensure quality.
• Drive platform performance, scalability, reliability and cost optimization.
• Support production deployments, issue resolution and root-cause analysis.
• Contribute to technology roadmaps and innovative solutions.

Skills

Databricks

PySpark

Apache Spark

Delta Lake

Lakehouse architecture

Advanced SQL

PL/SQL

ETL/ELT

Data Warehousing

Data modeling

CI/CD

Git

AWS

Tools

Hive

DataFrames

Spark SQL

Data quality

Job description

Tata Consultancy Services Limited in Boynton Beach, FL seeks a Senior Data Engineer to lead design and implementation of large-scale data pipelines. You will work with Databricks and PySpark to build scalable batch and real-time data solutions and mentor junior engineers.

Role requires 8+ years in data engineering, Delta Lake mastery, and strong SQL/PLSQL skills. On-site role with a focus on cost optimization and reliable data platforms.