Databricks Architect - Remote Lakehouse & Delta Expert

Tata Consultancy Services; United States; Hybrid; USD 130,000 - 170,000; Full time

Job summary

Tata Consultancy Services is seeking a Databricks Architect with over 5 years of data engineering experience, either based in Marlborough, MA or fully remote with travel required. The candidate should possess deep expertise in the Databricks Lakehouse Platform, be proficient in Python, PySpark, and SQL, and have strong cloud experience. Responsibilities include designing scalable data pipelines, ensuring data quality, and collaborating with functional teams. This position offers opportunities to work in a dynamic environment with cutting-edge technologies.

Qualifications

• 5+ years of hands-on data engineering experience, including 3 years in Databricks/Spark.
• Deep expertise in the Databricks Lakehouse Platform, involving Delta Lake and structured streaming.
• Expert-level proficiency in Python, PySpark, and advanced SQL skills.
• Strong understanding of data modeling and ETL/ELT design patterns.
• Experience with major cloud providers (Azure, AWS, GCP).

Responsibilities

• Design and deploy robust batch and streaming data pipelines using PySpark and Spark SQL.
• Implement data quality frameworks and ensure accuracy of Customer 360 data.
• Monitor and optimize Databricks jobs and Spark clusters for performance.
• Collaborate with cross-functional teams to deliver consumption-ready datasets.

Skills

Data engineering experience

Databricks expertise

Proficiency in Python and PySpark

Strong SQL skills

Cloud proficiency (Azure, AWS, GCP)

Data warehousing concepts

Software engineering practices

Data pipeline development

Data quality implementation

Performance optimization

Tools

Databricks

Terraform

AWS DevOps

GitHub Actions

Job description

Tata Consultancy Services is seeking a Databricks Architect with over 5 years of data engineering experience, either based in Marlborough, MA or fully remote with travel required. The candidate should possess deep expertise in the Databricks Lakehouse Platform, be proficient in Python, PySpark, and SQL, and have strong cloud experience. Responsibilities include designing scalable data pipelines, ensuring data quality, and collaborating with functional teams. This position offers opportunities to work in a dynamic environment with cutting-edge technologies.