Senior Databricks Data Engineer: Lakehouse & Streaming

Tata Consultancy Services Limited; Quincy (MA); On-site; USD 110,000 - 126,000; Full time

Job summary

Tata Consultancy Services Limited in Quincy, MA is seeking an experienced Engineer to lead data engineering initiatives using Databricks and Spark. The role focuses on building scalable data pipelines, streaming solutions, and lakehouse architectures to support analytics and decision making.

Candidates should have 8+ years in data engineering with Scala, PySpark, SQL, and hands-on Databricks implementation. Familiarity with DevOps, Azure services, and containerization is a plus.

Qualifications

• 8+ years of data engineering experience with Databricks and ETL/ELT data pipelines.
• SQL experience and processing pipelines.
• Hands-on Databricks with Scala and Spark (dataframes, notebooks, SQL).
• 8+ years in Scala/PySpark/SQL development and performance tuning.
• Databricks delta lake storage, Unity catalog, Autoloader and lakehouse architectures.
• Experience with Databricks serverless options, clusters sizing and job orchestration.
• Familiarity with Spark structured streaming.

Responsibilities

• Develop complex data streaming and transformations using Azure Databricks.
• Implement data warehouse/data lakehouse solutions.
• Monitor, debug and resolve issues in Databricks jobs.
• Troubleshoot DevOps pipelines, Azure, and AKS services.
• Provide ingestion, streaming and transformation solutions and development work.
• Work with PySpark, R, or Java in big data environments.
• Familiarity with containerization (Docker, Kubernetes) is a plus.

Skills

Scala

SQL

Education

Bachelor of Computer Science

Job description

Tata Consultancy Services Limited in Quincy, MA is seeking an experienced Engineer to lead data engineering initiatives using Databricks and Spark. The role focuses on building scalable data pipelines, streaming solutions, and lakehouse architectures to support analytics and decision making.

Candidates should have 8+ years in data engineering with Scala, PySpark, SQL, and hands-on Databricks implementation. Familiarity with DevOps, Azure services, and containerization is a plus.