ADF Databricks Pyspark Python Terraform

Tata Consultancy Services; Hyderabad; Presencial; INR 1.500.000 - 2.500.000; Jornada completa

Descripción de la vacante

Tata Consultancy Services (TCS) is hiring for a hands-on ADF Databricks PySpark Python Terraform specialist in Hyderabad. The role demands 6–9 years of IT experience and strong expertise in cloud data platforms.

You will design scalable ETL/ELT pipelines, build data lakes and warehouses, and implement CI/CD with Terraform in an Agile environment. Collaboration with stakeholders and adherence to data governance is essential.

Formación

• 6–9 years of IT experience is mandatory.
• Experience designing and building cloud-native data platforms is required.
• Strong understanding of data lake/data warehouse concepts and ETL/ELT processes must be shown.

Responsabilidades

• Design and develop scalable ETL/ELT pipelines using Azure Data Factory and Azure Databricks.
• Develop batch and near real-time data processing solutions using PySpark and Python.
• Build and maintain enterprise data lakes and data warehouse solutions.
• Implement data ingestion frameworks for structured, semi-structured, and unstructured data.
• Develop reusable data transformation and data quality frameworks.
• Provision and manage Azure infrastructure using Terraform.
• Optimize Databricks jobs, clusters, and Spark workloads for performance and cost efficiency.
• Implement CI/CD pipelines and deployment automation.
• Collaborate with architects, data analysts, and business stakeholders to gather requirements.
• Perform data validation, monitoring, troubleshooting, and production support.
• Ensure compliance with security, governance, and data management standards.

Conocimientos

Azure Databricks

ADF

PySpark

Python

Terraform

Spark tuning

CI/CD

Agile/Scrum

Data modeling

Data warehousing concepts

Educación

15 years of Full Time Education

Herramientas

Azure Databricks

Azure Data Factory

PySpark

Python

Terraform

Descripción del empleo

TCS Hiring for ADF Databricks Pyspark Python Terraform

Experience Range: - 06 To 09 Years (Mandatory)

(Note: Candidates below 6 years of IT experience shall not be considered)

JOB DESCRIPTION

Key Responsibilities:

• Design and develop scalable ETL/ELT pipelines using Azure Data Factory and Azure Databricks.
• Develop batch and near real-time data processing solutions using PySpark and Python.
• Build and maintain enterprise data lakes and data warehouse solutions.
• Implement data ingestion frameworks for structured, semi-structured, and unstructured data.
• Develop reusable data transformation and data quality frameworks.
• Provision and manage Azure infrastructure using Terraform.
• Optimize Databricks jobs, clusters, and Spark workloads for performance and cost efficiency.
• Implement CI/CD pipelines and deployment automation.
• Collaborate with architects, data analysts, and business stakeholders to gather requirements.
• Perform data validation, monitoring, troubleshooting, and production support.
• Ensure compliance with security, governance, and data management standards.

Desired Candidate Profile:

• Strong hands-on experience in Azure Databricks, ADF, PySpark, Python, and Terraform.
• Experience building scalable cloud-native data platforms.
• Strong understanding of data lake, data warehouse, and ETL concepts.
• Expertise in Spark performance tuning and optimization.
• Excellent problem-solving and analytical skills.
• Experience working in Agile/Scrum environments.
• Strong communication and stakeholder management skills.

Mandatory Technical Skills:

2) Programming & Data Engineering: Python, PySpark, SQL, Apache Spark, Data Modeling, ETL/ELT Development, Data Warehousing Concepts

15 years of Full Time Education