Walk-in | Pyspark Developer

Tata Consultancy Services; Chennai District; On-site; INR 1,400,000 - 2,000,000; Full time

Job summary

Tata Consultancy Services is seeking an experienced PySpark Developer to design, build, and maintain large-scale data processing solutions using Apache Spark and Python. The role focuses on data engineering, processing, and cloud-based data platforms to optimize performance and ensure data quality.

The ideal candidate will work across data ingestion, transformation, and integration from various sources, collaborating with cross-functional teams and leveraging Spark streaming with Kafka for

Qualifications

• Strong experience with PySpark, Spark, and Python in data engineering.
• Experience designing data processing pipelines and optimizing performance.
• Proficient in cloud platforms (AWS, Azure, or GCP).
• Hands-on knowledge of DataFrames, Spark SQL, and Spark Streaming with Kafka for real-time ingestion.
• Understanding distributed computing concepts and multithreading in PySpark.
• Proficiency in Python, Java, or Scala with good communication skills.

Responsibilities

• Design, develop, and maintain large-scale data processing solutions using PySpark and Python.
• Develop and optimize data pipelines in cloud-based environments.
• Integrate data from relational, NoSQL, and cloud storage sources.
• Implement data transformation logic and ensure data quality across systems.
• Collaborate with cross-functional teams to prioritize requirements and provide technical guidance.

Skills

PySpark

Data processing

Cloud platforms

DataFrames & Spark SQL

Spark Streaming

Multithreading concepts

Programming languages

Communication

Job description

Job Summary

An experienced PySpark Developer / Leads to design, develop, and maintain large-scale data processing solutions using Apache Spark and Python. The ideal candidate should have a strong background in data engineering, data processing, and cloud-based data platforms, big data ecosystems and performance optimization

Responsibilities

• Data Processing: Design, develop, and maintain data processing solutions using PySpark, Apache Spark, and Python.
• Data Pipeline Development: Develop and optimize data pipelines using PySpark, Apache Spark, and cloud-based data platforms.
• Data Integration: Integrate data from various sources, including relational databases, NoSQL databases, and cloud storage.
• Data Transformation: Develop and implement data transformation logic using PySpark, Apache Spark, and Python.
• Collaboration: Work with cross-functional teams to identify and prioritize project requirements, provide technical guidance, and ensure data quality.

Required Skills

• PySpark: In-depth knowledge of PySpark, Apache Spark, and Python.
• Data Processing: Strong understanding of data processing concepts, including data ingestion, data transformation, and data storage.
• Cloud Experience: Experience with cloud-based data platforms, including AWS, Azure, or Google Cloud.
• Expertise in DataFrames & Spark SQL, Spark Streaming with Apache Kafka real-time data ingestion pipelines
• Very good conceptual understanding of Multithreading, distributed computing concepts of Pyspark
• Programming: Proficiency in programming languages, including Python, Java, or Scala.
• Communication: Excellent communication and collaboration skills.