Contractor - Python & PySpark Developer

Vivantify; Hyderabad; On-site; INR 1,800,000 - 3,200,000; Full time

Benefits offered by this job

Competitive salary

Growth opportunities

Collaborative environment

Job summary

Vivantify is seeking a Python and PySpark Developer to design, build, and optimize data pipelines across Pan India. The role emphasizes PySpark ETL, Spark SQL, DataFrames, Delta Lake, and robust testing using PyTest.

You will leverage GenAI tools to enhance development, debugging, documentation, and code quality while ensuring data privacy and secure production practices. The ideal candidate has 5–7 years of hands-on experience, strong scripting and logging capabilities, and a proactive approach

Qualifications

• 5–7 years of experience in Python and PySpark development.
• Hands-on expertise in ETL processing, Spark, Delta Lake, testing, and production troubleshooting.
• Experience using GenAI and AI coding assistants for development, analysis, testing, documentation, and code quality improvement.

Responsibilities

• Maintain and enhance PySpark ETL jobs and data processing pipelines.
• Develop and troubleshoot Spark DataFrame, Spark SQL, and Delta Lake processing.
• Implement file ingestion for CSV, XML, and Hyper extracts.
• Work with SQL-to-PySpark integration and reporting-date handling.
• Develop and execute PyTest-based unit and integration tests.
• Troubleshoot production issues using Python logging and exception handling.
• Use AI coding assistants to accelerate development, debugging, testing, review, and documentation.
• Analyze legacy code and support root-cause analysis with AI-assisted techniques.

Skills

Python

PySpark

Spark DataFrames

Spark SQL

Delta Lake

ETL development

Python debugging

SQL-to-PySpark

File ingestion

Python packaging

PyTest

Pylint

GitHub Copilot

ChatGPT

GenAI-assisted code analysis

Prompt engineering

Responsible AI

Tools

AKEG

Job description

Location: Pan India

Experience: 5–7 Years

About the Role

We are looking for an experienced Python and PySpark Developer with strong hands‑on expertise in PySpark ETL, Python, Spark SQL, DataFrames, and Delta Lake. The role focuses on developing, enhancing, testing, and supporting data processing pipelines, with an emphasis on using GenAI tools to improve development, debugging, testing, documentation, and code quality.

Job Description

The role involves maintaining and enhancing PySpark ETL jobs and troubleshooting data processing using Spark DataFrames, Spark SQL, and Delta Lake. You will work with file ingestion processes, including CSV, XML, and Hyper extracts, along with reporting‑date handling and SQL‑to‑PySpark integration.

You will also develop and execute PyTest‑based unit and integration tests and troubleshoot production issues using Python logging and exception handling. The position requires effective use of AI coding assistants such as GitHub Copilot, ChatGPT, or similar tools for code development, analysis, testing, debugging, documentation, and code reviews.

The role also emphasizes responsible AI usage, including data privacy, secure handling of production data, validation of AI‑generated outputs, and awareness of prompt engineering for technical tasks.

Key Responsibilities

• Maintain and enhance PySpark ETL jobs and data processing pipelines
• Develop and troubleshoot Spark DataFrame, Spark SQL, and Delta Lake processing
• Implement file ingestion for CSV, XML, and Hyper extracts
• Work with SQL‑to‑PySpark integration and reporting‑date handling
• Develop and execute PyTest‑based unit and integration tests
• Troubleshoot production issues using Python logging and exception handling
• Use AI coding assistants to accelerate development, debugging, testing, code review, and documentation
• Analyze legacy code, perform impact assessments, and support root‑cause analysis using AI‑assisted techniques

Required Skills

• Python
• PySpark
• Spark DataFrames and Spark SQL
• Delta Lake
• ETL development
• Python debugging, logging, and exception handling
• SQL‑to‑PySpark integration
• File ingestion and processing
• Python packaging and runtime dependencies
• PyTest‑based testing
• Code quality tools such as Pylint
• AI coding assistants such as GitHub Copilot or ChatGPT
• GenAI‑assisted code analysis, testing, debugging, and documentation
• Prompt engineering for technical tasks
• Responsible AI practices and production data privacy

Good to Have

• Exposure to ML/AI concepts
• Embeddings
• Vector search
• RAG
• LLM‑based knowledge assistants for internal documentation and search

Mandatory Skills

• Python

Candidate Profile

The ideal candidate should have 5–7 years of experience in Python and PySpark development, with strong hands‑on expertise in ETL processing, Spark, Delta Lake, testing, and production troubleshooting. The candidate should also be comfortable using GenAI and AI coding assistants for development, analysis, testing, documentation, and code‑quality improvement, with strong communication skills.

Benefits Package

• Competitive salary and benefits package
• Opportunities for professional growth and development
• A collaborative and inclusive work environment
• The opportunity to work on exciting and challenging projects with leading clients