Tcs Hiring For Pyspark Data Engineer

Tata Consultancy Services; Chennai District; On-site; INR 1,200,000 - 2,400,000; Full time

Job summary

Tata Consultancy Services in Chennai is seeking a PySpark Data Engineer to design, develop, and optimize large-scale data processing pipelines using PySpark and Spark ecosystem.

You will work with Data Scientists and Analysts to ingest, transform, and analyze data, build scalable ETL/ELT solutions, and ensure data quality across sources.

Qualifications

• Strong hands-on experience in PySpark development.
• Expertise in Spark SQL, DataFrames, and Spark optimization techniques.
• Experience with Hadoop ecosystem (Hive, HDFS, Kafka).
• Good knowledge of SQL and database concepts.
• Experience in ETL pipeline development.
• Knowledge of Azure Databricks, Azure Data Factory, or AWS EMR is preferred.
• Experience with Git, CI/CD, and Agile methodologies.

Responsibilities

• Develop and optimize data pipelines using PySpark and Apache Spark.
• Perform data ingestion, transformation, and processing of large datasets.
• Design scalable ETL/ELT solutions for data warehousing and analytics.
• Work with distributed computing frameworks and big data technologies.
• Integrate data from multiple sources and ensure data quality.
• Collaborate with Data Scientists, Analysts, and Business teams.
• Troubleshoot performance issues and optimize Spark jobs.
• Follow best practices for data governance, security, and code quality.

Skills

PySpark development

Spark SQL DataFrames

SQL and database concepts

ETL pipeline development

CI/CD Agile methodologies

Tools

Hadoop Hive HDFS Kafka

Azure Databricks

Azure Data Factory

AWS EMR

Git

CI/CD

Job description

Job Description PySpark Data Engineer

Job Title: PySpark Data Engineer Experience: 5-10 Years Location: Chennai

Job Summary

We are looking for a skilled PySpark Data Engineer to design, develop, and optimize large-scale data processing pipelines. The ideal candidate should have strong expertise in PySpark, Spark ecosystem, SQL, ETL development, and cloud/big data technologies.

Key Responsibilities

• Develop and optimize data pipelines using PySpark and Apache Spark.
• Perform data ingestion, transformation, and processing of large datasets.
• Design scalable ETL/ELT solutions for data warehousing and analytics.
• Work with distributed computing frameworks and big data technologies.
• Integrate data from multiple sources and ensure data quality.
• Collaborate with Data Scientists, Analysts, and Business teams.
• Troubleshoot performance issues and optimize Spark jobs.
• Follow best practices for data governance, security, and code quality.

Required Skills

• Strong hands-on experience in PySpark development.
• Expertise in Spark SQL, DataFrames, and Spark optimization techniques.
• Experience with Hadoop ecosystem (Hive, HDFS, Kafka).
• Good knowledge of SQL and database concepts.
• Experience in ETL pipeline development.
• Knowledge of Azure Databricks, Azure Data Factory, or AWS EMR is preferred.
• Experience with Git, CI/CD, and Agile methodologies.