Pyspark Developer

Weekday AI (YC W21); Bengaluru; On-site; INR 2,000,000 - 3,000,000; Full time

Job summary

Weekday AI (YC W21) is seeking an experienced PySpark Developer to design and optimize scalable data pipelines in Bengaluru, Chennai, and across India. The role requires strong Python, Spark expertise and hands-on data engineering experience with ETL/ELT processes and data quality controls.

The ideal candidate will lead data pipeline development, collaborate with engineers and analysts, and drive performance improvements while handling large-scale datasets and production deployments.

Qualifications

• 8–13 years of experience in data engineering or related roles.
• Strong hands-on PySpark, Spark, and Python.
• Proven experience building scalable data pipelines.
• Knowledge of ETL/ELT and data quality practices.
• Experience with SQL and data warehousing concepts.
• Familiarity with Git, debugging, testing, and production support.

Responsibilities

• Design, develop, and maintain scalable data pipelines using PySpark and Python.
• Develop complex data transformations with Spark DataFrames and Spark SQL.
• Build reliable ETL/ELT processes for large data volumes and multiple sources.
• Ensure data quality, lineage, and monitoring across pipelines.
• Collaborate with data engineers, analysts, and architects; mentor juniors.

Skills

PySpark

Apache Spark

Python

Data pipelines

ETL/ELT

Distributed computing

SQL

Data warehousing

Git

Testing & debugging

Tools

Databricks

Delta Lake

Snowflake

Airflow

Job description

Job Description:

This role is for one of Weekday’s clients
Salary range: Rs 2000000 - Rs 3000000 (ie INR 20 - 30 LPA)

Min Experience: 8+ years
Location: Bengaluru, Chennai, Tamil Nadu, India
JobType: full-time

We are looking for an experienced PySpark Developer with 8–13 years of experience in data engineering, Python programming, and large-scale data processing. The ideal candidate will have strong expertise in developing, optimizing, and maintaining scalable data pipelines using PySpark and Apache Spark. This role requires hands‑on experience in processing structured and semi-structured data, implementing ETL/ELT workflows, and building reliable data solutions that support business intelligence, analytics, and data-driven decision‑making.

The candidate will collaborate with data engineers, architects, analysts, and other technical stakeholders to design efficient data processing solutions, improve pipeline performance, and ensure data quality across the organization.

Key Responsibilities

1. Data Pipeline Development

• Design, develop, and maintain scalable, high-performance data pipelines using PySpark and Python.
• Build robust ETL/ELT workflows to ingest, transform, validate, and load large volumes of data from multiple sources.
• Develop batch and, where required, streaming data processing solutions to support business and analytical requirements.
• Implement reusable data processing frameworks and ensure pipelines are reliable, maintainable, and scalable.
• Integrate data from databases, APIs, cloud storage, and enterprise data platforms.

2. PySpark Development and Optimization

• Develop complex data transformations using PySpark DataFrames, Spark SQL, and distributed processing techniques.
• Optimize Spark jobs by analyzing execution plans, partitioning strategies, caching, and resource utilization.
• Troubleshoot performance bottlenecks, resolve processing failures, and improve job execution efficiency.
• Work with large datasets while ensuring efficient memory management and distributed computing performance.
• Apply best practices for handling joins, aggregations, data skew, and incremental data processing.

3. Python Programming

• Write clean, modular, reusable, and well-documented Python code for data engineering applications.
• Develop utilities, validation scripts, and automation solutions to streamline data workflows.
• Implement exception handling, logging, testing, and debugging practices to improve code quality.
• Collaborate with engineering teams to maintain coding standards and version control practices.

4. Data Quality and Reliability

• Implement data validation, reconciliation, and monitoring mechanisms across pipelines.
• Identify and resolve data inconsistencies, duplication, and transformation errors.
• Maintain technical documentation, data lineage, and operational procedures.
• Support production deployments, incident resolution, and continuous improvement initiatives.

5. Collaboration and Technical Leadership

• Work closely with business stakeholders and technical teams to translate requirements into scalable data solutions.
• Participate in architecture discussions, code reviews, technical design sessions, and development planning.
• Mentor junior developers and contribute to engineering best practices.

Required Skills and Qualifications

• 8–13 years of overall experience in software development, data engineering, or related technical roles.
• Strong hands‑on expertise in PySpark, Apache Spark, and Python.
• Proven experience designing and maintaining scalable data pipelines and ETL/ELT processes.
• Strong understanding of distributed computing, data partitioning, transformations, and performance optimization.
• Experience working with SQL, relational databases, and large-scale structured or semi-structured datasets.
• Familiarity with data warehousing concepts, data modeling, and data quality practices.
• Experience with Git, debugging, testing, and production support.

Preferred Qualifications

• Experience with cloud data platforms such as AWS, Microsoft Azure, or Google Cloud.
• Exposure to platforms such as Databricks, Delta Lake, Snowflake, or cloud-based data lakes.
• Familiarity with orchestration tools such as Apache Airflow and CI/CD practices.
• Knowledge of Kafka or other streaming technologies is an advantage.