Pyspark Developer

Tata Consultancy Services; India; Presencial; INR 900 000 - 1 400 000; Tempo integral

Resumo da oferta

Tata Consultancy Services in India is seeking an experienced PySpark Developer to design, develop, and maintain large-scale data processing solutions using PySpark, Spark, and cloud data platforms. The role emphasizes data engineering, data processing, and performance optimization.

The ideal candidate will collaborate with cross-functional teams to ensure data quality and scalable pipelines, leveraging Spark SQL, DataFrames, and streaming technologies while optimizing performance across

Qualificações

• Proficient in PySpark, Spark, and Python for large-scale data processing.
• Experience with data ingestion, transformation, and storage pipelines.
• Experience with cloud-based data platforms (AWS/Azure/GCP).
• Strong understanding of DataFrames & Spark SQL; real-time data ingestion with Spark Streaming is a plus.
• Knowledge of multithreading and distributed computing concepts in PySpark.
• Proficient in Python/Java/Scala programming.
• Excellent communication and collaboration skills.

Responsabilidades

• Data Processing: Design, develop, and maintain data processing solutions using PySpark, Apache Spark, and Python.
• Data Pipeline Development: Develop and optimize data pipelines using PySpark, Apache Spark, and cloud-based data platforms.
• Data Integration: Integrate data from relational, NoSQL sources, and cloud storage.
• Data Transformation: Implement data transformation logic using PySpark and Python.
• Collaboration: Work with cross-functional teams to define requirements and ensure data quality.

Conhecimentos

PySpark

Data Processing

Cloud Experience

Spark SQL & DataFrames

Multithreading concepts

Programming (Python/Java/Scala)

Communication

Ferramentas

Apache Spark

PySpark

Kafka

Hadoop

Descrição da oferta de emprego

Job Title

PySpark Developers

Experience

5-10 years

Job Summary

An experienced PySpark Developer / Leads to design, develop, and maintain large-scale data processing solutions using Apache Spark and Python. The ideal candidate should have a strong background in data engineering, data processing, and cloud-based data platforms, big data ecosystems and performance optimization

Responsibilities

• Data Processing: Design, develop, and maintain data processing solutions using PySpark, Apache Spark, and Python.
• Data Pipeline Development: Develop and optimize data pipelines using PySpark, Apache Spark, and cloud-based data platforms.
• Data Integration: Integrate data from various sources, including relational databases, NoSQL databases, and cloud storage.
• Data Transformation: Develop and implement data transformation logic using PySpark, Apache Spark, and Python.
• Collaboration: Work with cross-functional teams to identify and prioritize project requirements, provide technical guidance, and ensure data quality.

Required Skills

• PySpark: In-depth knowledge of PySpark, Apache Spark, and Python.
• Data Processing: Strong understanding of data processing concepts, including data ingestion, data transformation, and data storage.
• Cloud Experience: Experience with cloud-based data platforms, including AWS, Azure, or Google Cloud.
• Expertise in DataFrames & Spark SQL, Spark Streaming with Apache Kafka real-time data ingestion pipelines
• Very good conceptual understanding of Multithreading, distributed computing concepts of Pyspark
• Programming: Proficiency in programming languages, including Python, Java, or Scala.
• Communication: Excellent communication and collaboration skills.

Preferred Skills

• Apache Spark Certifications: Relevant certifications, such as Apache Spark Certification or Cloudera Certified Spark Developer.
• Big Data: Experience with big data technologies, including Hadoop, HBase, or Cassandra.
• Machine Learning: Experience with machine learning frameworks, including TensorFlow, PyTorch, or Scikit-learn.
• Data Science: Experience with data science tools, including Jupyter Notebook, Pandas, or NumPy