Python Pyspark Developer

V2 Solutions; Ernakulam; On-site; INR 1,800,000 - 3,000,000; Full time

Job summary

V2 Solutions is seeking an experienced data engineer to design and implement big data pipelines using PySpark on AWS EMR. You will collaborate with cross-functional teams to deliver scalable ETL solutions and ensure data quality across large datasets.

The role requires 5-10 years of Python/PySpark experience, strong Django skills for web applications, and proven ability to optimize performance and reliability in data workflows.

Qualifications

• 5-10 years of experience building big data apps with Python/PySpark on AWS.
• Strong Django framework experience for web apps.
• Experience designing complex ETL pipelines for large datasets.

Responsibilities

• Design, develop, test, deploy and maintain large-scale data pipelines using PySpark on AWS EMR.
• Create scalable ETL processes to extract insights from diverse data sources.
• Troubleshoot data quality, performance, and reliability issues.

Skills

PySpark

AWS EMR

Django

ETL design

Spark SQL

Data quality

Performance tuning

Troubleshooting

Cross-functional collaboration

Tools

Python

Job description

Roles and Responsibilities

• Design, develop, test, deploy and maintain large-scale data processing pipelines using PySpark on AWS EMR. Collaborate with cross-functional teams to gather requirements and deliver high-quality solutions that meet business needs.
• Develop scalable and efficient ETL processes to extract insights from various sources of structured and unstructured data.
• Troubleshoot issues related to data quality, performance optimization, and system reliability.

Job Requirements

• 5-10 years of experience in developing big data applications using Python/PySpark on AWS platform.
• Strong understanding of Django framework for building web applications.
• Experience with designing complex ETL processes for large-scale datasets.
• Proficiency in working with Spark SQL for querying massive datasets.