Contractor - Python & PySpark Developer

Vivantify Technology Solutions India Pvt. Ltd.; Hyderabad; Remote; INR 1,200,000 - 1,900,000; Full time

Benefits offered by this job

Competitive salary

Growth opportunities

Inclusive environment

Challenging projects

Job summary

Vivantify Technology Solutions India Pvt. Ltd. is seeking an experienced Python and PySpark Developer to design, develop, test, and optimize PySpark ETL pipelines using Spark DataFrames, Spark SQL, and Delta Lake.

The role emphasizes AI-assisted coding to improve quality and speed. You will handle file ingestion (CSV/XML/Hyper extracts), implement testing with PyTest, and troubleshoot production data processing.

Qualifications

• 5–7 years of hands-on Python and PySpark development experience.
• Experience building PySpark ETL pipelines with Delta Lake.

Responsibilities

• Maintain and enhance PySpark ETL jobs and data processing pipelines.
• Develop and troubleshoot Spark DataFrame, Spark SQL, and Delta Lake processing.
• Implement file ingestion for CSV, XML, and Hyper extracts.
• Work with SQL-to-PySpark integration and reporting-date handling.
• Develop and execute PyTest-based unit and integration tests.
• Troubleshoot production issues using Python logging and exception handling.
• Use AI coding assistants to accelerate development, debugging, testing, code review, and documentation.
• Analyze legacy code and perform impact assessments with AI-assisted techniques.

Skills

Python

PySpark

Spark DataFrames

Spark SQL

Delta Lake

ETL development

PyTest testing

AI coding assistants

GenAI assisted coding

Prompt engineering

Data privacy

Tools

GitHub Copilot

ChatGPT

Job description

About The Role

We are looking for an experienced

Location: Pan India

Experience: 5–7 Years

About The Role

We are looking for an experienced Python and PySpark Developer with strong hands-on expertise in PySpark ETL, Python, Spark SQL, DataFrames, and Delta Lake. The role focuses on developing, enhancing, testing, and supporting data processing pipelines, with an emphasis on using GenAI tools to improve development, debugging, testing, documentation, and code quality.

Job Description

The role involves maintaining and enhancing PySpark ETL jobs and troubleshooting data processing using Spark DataFrames, Spark SQL, and Delta Lake. You will work with file ingestion processes, including CSV, XML, and Hyper extracts, along with reporting-date handling and SQL-to-PySpark integration.

You will also develop and execute PyTest-based unit and integration tests and troubleshoot production issues using Python logging and exception handling. The position requires effective use of AI coding assistants such as GitHub Copilot, ChatGPT, or similar tools for code development, analysis, testing, debugging, documentation, and code reviews.

The role also emphasizes responsible AI usage, including data privacy, secure handling of production data, validation of AI-generated outputs, and awareness of prompt engineering for technical tasks.

Key Responsibilities

• Maintain and enhance PySpark ETL jobs and data processing pipelines
• Develop and troubleshoot Spark DataFrame, Spark SQL, and Delta Lake processing
• Implement file ingestion for CSV, XML, and Hyper extracts
• Work with SQL-to-PySpark integration and reporting-date handling
• Develop and execute PyTest-based unit and integration tests
• Troubleshoot production issues using Python logging and exception handling
• Use AI coding assistants to accelerate development, debugging, testing, code review, and documentation
• Analyze legacy code, perform impact assessments, and support root-cause analysis using AI-assisted techniques
  
  
  

Required Skills

• Python
• PySpark
• Spark DataFrames and Spark SQL
• Delta Lake
• ETL development
• Python debugging, logging, and exception handling
• SQL-to-PySpark integration
• File ingestion and processing
• Python packaging and runtime dependencies
• PyTest-based testing
• Code quality tools such as Pylint
• AI coding assistants such as GitHub Copilot or ChatGPT
• GenAI-assisted code analysis, testing, debugging, and documentation
• Prompt engineering for technical tasks
• Responsible AI practices and production data privacy
  
  
  

Good to Have

• Exposure to ML/AI concepts
• Embeddings
• Vector search
• RAG
• LLM-based knowledge assistants for internal documentation and search
  
  
  

Mandatory Skills

• Python
  
  
  

Candidate Profile

The ideal candidate should have 5–7 years of experience in Python and PySpark development, with strong hands-on expertise in ETL processing, Spark, Delta Lake, testing, and production troubleshooting. The candidate should also be comfortable using GenAI and AI coding assistants for development, analysis, testing, documentation, and code-quality improvement, with strong communication skills.

Benefits Package

• Competitive salary and benefits package
• Opportunities for professional growth and development
• A collaborative and inclusive work environment
• The opportunity to work on exciting and challenging projects with leading clients