Databricks Lead

Tata Consultancy Services; Dadri, Chennai District; On-site; INR 700,000 - 1,200,000; Full time

Job summary

Tata Consultancy Services in India is seeking a Data Engineer to design, develop, and optimize scalable data pipelines and analytics solutions using Databricks, PySpark, Python, PL/SQL, and Azure Data Factory. You will work with large-scale data processing, ETL/ELT frameworks, cloud platforms, and data warehouses, collaborating with analysts and data scientists to deliver reliable, governed data workflows.

The role emphasizes hands-on execution, code reviews, and production deployments in a

Qualifications

• 410 years of experience in Data Engineering or related roles.
• Strong hands-on experience with Databricks and PySpark.
• Proficiency in Python programming for data processing and automation.
• Strong experience in PL/SQL development and performance tuning.
• Experience with Azure Data Factory (ADF) for data integration and orchestration.
• Knowledge of data warehousing concepts and ETL/ELT processes.
• Experience working with SQL databases and large datasets.
• Understanding of Azure cloud services and data platform architecture.
• Strong analytical and problem-solving skills.

Responsibilities

• Design, develop, and maintain data pipelines using Databricks, PySpark, and Azure Data Factory (ADF).
• Build scalable ETL/ELT solutions for processing structured and unstructured data.
• Develop and optimize data transformation workflows using Python and PySpark.
• Write, maintain, and tune complex PL/SQL queries, stored procedures, and database objects.
• Integrate data from multiple sources into enterprise data warehouses and data lakes.
• Monitor and troubleshoot data pipeline performance, ensuring reliability and data quality.
• Collaborate with business stakeholders, analysts, and data scientists to understand data requirements.
• Implement data governance, security, and best practices across data platforms.
• Support cloud-based data migration and modernization initiatives.
• Participate in code reviews, testing, and production deployments.

Tools

Databricks

PySpark

Python

PL/SQL

Azure Data Factory (ADF)

Job description

We are looking for a skilled Data Engineer with expertise in Databricks, PySpark, Python, PL/SQL, and Azure Data Factory (ADF) to design, develop, and optimize scalable data pipelines and analytics solutions. The ideal candidate will have experience working with large-scale data processing, ETL/ELT frameworks, cloud-based data platforms, and data warehousing solutions.

Key Responsibilities

• Design, develop, and maintain data pipelines using Databricks, PySpark, and Azure Data Factory (ADF).
• Build scalable ETL/ELT solutions for processing structured and unstructured data.
• Develop and optimize data transformation workflows using Python and PySpark.
• Write, maintain, and tune complex PL/SQL queries, stored procedures, and database objects.
• Integrate data from multiple sources into enterprise data warehouses and data lakes.
• Monitor and troubleshoot data pipeline performance, ensuring reliability and data quality.
• Collaborate with business stakeholders, analysts, and data scientists to understand data requirements.
• Implement data governance, security, and best practices across data platforms.
• Support cloud-based data migration and modernization initiatives.
• Participate in code reviews, testing, and production deployments.

Required Skills

• 410 years of experience in Data Engineering or related roles.
• Strong hands-on experience with Databricks and PySpark.
• Proficiency in Python programming for data processing and automation.
• Strong experience in PL/SQL development and performance tuning.
• Experience with Azure Data Factory (ADF) for data integration and orchestration.
• Knowledge of data warehousing concepts and ETL/ELT processes.
• Experience working with SQL databases and large datasets.
• Understanding of Azure cloud services and data platform architecture.
• Strong analytical and problem-solving skills.

Preferred Skills

• Experience with Azure Data Lake Storage (ADLS).
• Exposure to Delta Lake, Azure Synapse Analytics, or Snowflake.
• Experience with CI/CD and DevOps practices.
• Knowledge of Data Governance and Data Quality frameworks.
• Azure certifications are a plus.