Databricks Lead

Tata Consultancy Services; Pune District, Bengaluru; On-site; INR 2,500,000 - 6,000,000; Full time

Job summary

Tata Consultancy Services in Pune, India invites a Databricks Lead/Senior Developer to architect and optimize data pipelines on the Databricks platform. You will apply PySpark, Spark SQL, and Azure event hubs/grid to build scalable solutions with security and governance in mind.

Responsibilities include designing processes, mentoring juniors, optimizing queries, and staying current with Databricks and Azure innovations to deliver high-quality data solutions for complex workloads.

Qualifications

• Proven experience as a Databricks Lead or similar role.
• Strong expertise in Databricks, PySpark, and Spark SQL.
• Experience with Azure services, including Azure Event Hubs and Azure Event Grid.
• Proficiency in designing and implementing data processing pipelines.
• Knowledge of data warehousing concepts and ETL processes.
• Familiarity with data security and compliance best practices.
• Excellent problem-solving and analytical skills.
• Strong communication and collaboration skills.

Responsibilities

• Design and architect scalable data processing solutions using Databricks and PySpark (Spark Streaming not DLT).
• Develop and maintain data pipelines, ensuring data quality and reliability.
• Implement data orchestration workflows using Azure Event Hubs and Azure Event Grid.
• Collaborate with data engineers, data scientists, and other stakeholders to understand data requirements and deliver solutions.
• Optimize and tune Spark queries for performance and efficiency.
• Ensure data security and compliance with industry standards and best practices.
• Provide technical guidance and mentorship to junior team members.
• Stay updated with the latest trends and advancements in Databricks, PySpark, and Azure services.

Skills

Databricks

PySpark

Spark SQL

Azure Event Hubs

Azure Event Grid

Data pipelines

Data security

Communication

Tools

Azure Data Factory

Docker

Kubernetes

Job description

Job Title: Databricks Lead / Senior Developer

Job Description:

We are seeking a highly skilled and experienced Databricks Lead/Senior Developer to join our team. The ideal candidate will possess strong expertise in Databricks, PySpark, and orchestration using Azure services such as Azure Event Hubs and Azure Event Grid. As a Databricks Lead, you will be responsible for designing, implementing, and optimizing data processing pipelines and solutions on the Databricks platform.

Key Responsibilities:

• Design and architect scalable data processing solutions using Databricks and PySpark (Spark Streaming not DLT).
• Develop and maintain data pipelines, ensuring data quality and reliability.
• Implement data orchestration workflows using Azure Event Hubs and Azure Event Grid.
• Collaborate with data engineers, data scientists, and other stakeholders to understand data requirements and deliver solutions.
• Optimize and tune Spark queries for performance and efficiency.
• Ensure data security and compliance with industry standards and best practices.
• Provide technical guidance and mentorship to junior team members.
• Stay updated with the latest trends and advancements in Databricks, PySpark, and Azure services.

Qualifications:

• Proven experience as a Databricks Lead or similar role.
• Strong expertise in Databricks, PySpark, and Spark SQL.
• Experience with Azure services, including Azure Event Hubs and Azure Event Grid.
• Proficiency in designing and implementing data processing pipelines.
• Knowledge of data warehousing concepts and ETL processes.
• Familiarity with data security and compliance best practices.
• Excellent problem-solving and analytical skills.
• Strong communication and collaboration skills.

Nicety To Have:

• Experience with other Azure services such as Azure Data Factory
• Certification in Databricks & Azure.
• Experience with containerization and orchestration tools like Docker and Kubernetes will add additional benefit.