Pyspark Developer

Purview Services; Mumbai; On-site; INR 1,200,000 - 1,800,000; Full time

Job summary

Purview Services in Mumbai, India is hiring for a Data Engineer to design, develop, and deploy large-scale data pipelines using PySpark and Spark. You will collaborate with cross-functional teams to understand business requirements and craft scalable solutions.

The role emphasizes writing clean, secure, and robust code, resolving complex data-processing issues, and participating in code reviews to improve the overall quality of the data platform. Prior experience with Agile methods is valued.

Qualifications

• Proficient in Python and its ecosystem.
• Experience with PySpark and Spark is required.
• Knowledge of Hadoop or Flink is a plus.
• Strong problem-solving and attention to detail.
• Ability to work in a team and communicate with stakeholders.
• Understanding of software development principles and patterns.
• Familiar with Agile development and Git.

Responsibilities

• Design, develop, and deploy large-scale data pipelines using PySpark.
• Collaborate with cross-functional teams to identify requirements and design solutions.
• Develop and maintain high-quality, secure, and reliable code.
• Troubleshoot and resolve data processing and pipeline issues.
• Participate in code reviews and improve codebase quality.
• Keep up-to-date with industry trends in big data processing.

Skills

Python

PySpark

Spark

Problem solving

Team collaboration

Software design principles

Agile methodologies

Git

Communication with stakeholders

Tools

Hadoop

Flink

Job description

Roles and Responsibility

• Design, develop, and deploy large-scale data processing pipelines using PySpark and Spark.
• Collaborate with cross-functional teams to identify business requirements and design solutions.
• Develop and maintain high-quality code that meets performance, security, and reliability standards.
• Troubleshoot and resolve complex technical issues related to data processing and pipeline failures.
• Participate in code reviews and contribute to the improvement of the overall codebase quality.
• Stay up-to-date with industry trends and emerging technologies in big data processing.

Job Requirements

• Strong proficiency in Python programming language and its ecosystem.
• Experience with PySpark and Spark is required; knowledge of other big data frameworks like Hadoop or Flink is a plus.
• Excellent problem-solving skills and attention to detail.
• Ability to work collaboratively in a team environment and communicate effectively with stakeholders.
• Strong understanding of software development principles, patterns, and practices.
• Familiarity with agile development methodologies and version control systems like Git.