Diversity Opportunities

Persistent Systems; Bengaluru; On-site; INR 2,800,000 - 4,600,000; Full time

Benefits offered by this job

Competitive salary

Company-sponsored education

Flexible work hours

Long service awards

Insurance coverage

Job summary

Persistent Systems is hiring a Python Data Engineer in Bengaluru for an in-person drive on 31st October 2026. You will build scalable data pipelines using Python, PySpark, SQL, and Airflow across cloud platforms, with emphasis on production-grade reliability.

The role requires 5-10 years of data engineering experience, strong CI/CD exposure with GitHub Actions or Harness, and hands-on work with GCP services like BigQuery and Dataproc. Collaborative work across cross-functional teams is expected.

Qualifications

• 5-10 years of experience in Data Engineering, ETL Development, or Big Data Engineering.
• Strong hands-on programming experience with Python.
• Extensive experience working with Apache Spark and PySpark.
• Strong SQL expertise with complex query development and data transformation techniques.
• Experience designing and implementing production-grade ETL/ELT pipelines.
• Strong understanding of distributed computing concepts and Spark architecture.
• Experience tuning Spark workloads for performance and scalability.
• Hands-on experience processing large-scale structured and semi-structured datasets.
• Experience working with cloud data platforms, preferably Google Cloud Platform (GCP).
• Knowledge of BigQuery, Dataproc, Apache Beam, and related GCP services is highly desirable.
• Experience developing and consuming REST APIs for data integration.
• Strong experience using Git/GitHub for version control, collaborative development, and code reviews.
• Practical experience with GitHub Actions, Harness, or similar CI/CD platforms.
• Experience implementing automated testing frameworks and deployment pipelines.
• Hands-on experience with Apache Airflow or equivalent workflow orchestration platforms.
• Strong understanding of logging, monitoring, alerting, error handling, and operational support processes.
• Experience troubleshooting production issues and maintaining highly available data pipelines.
• Familiarity with software engineering best practices, design patterns, and clean coding principles.

Responsibilities

• Design, develop, and maintain scalable Python and PySpark-based data pipelines for batch and large-scale data processing.
• Build reusable frameworks and components for data ingestion, transformation, validation, enrichment, and publishing.
• Process structured and semi-structured datasets from databases, APIs, cloud platforms, files, and enterprise applications.
• Develop, optimize, and maintain complex SQL queries and Spark transformations for high-volume datasets.
• Implement robust data-quality frameworks, reconciliation checks, exception handling, monitoring, and logging mechanisms.
• Develop modular, maintainable, testable, and reusable Python/PySpark code following engineering best practices.
• Manage source code repositories using GitHub, including branching, pull requests, code reviews, and release workflows.
• Build and maintain CI/CD pipelines using GitHub Actions, Harness, or similar automation platforms.
• Design and manage workflow orchestration using Apache Airflow and enterprise scheduling solutions.
• Optimize Spark jobs for performance, scalability, reliability, and cost efficiency.
• Troubleshoot data pipeline failures, ETL issues, data-quality problems, and production incidents.
• Participate in code reviews, testing activities, deployment processes, and production support.
• Collaborate with Data Architects, Data Analysts, Platform Engineers, DevOps teams, and Product Owners to deliver business solutions.
• Contribute to data platform modernization and continuous improvement initiatives.
• Ensure adherence to security, governance, and data engineering standards.

Skills

Python

Apache Spark

PySpark

SQL

Airflow

CI/CD

GitHub

GCP

BigQuery

Dataproc

Tools

GitHub Actions

Harness

Dataproc

BigQuery

Job description

We are conducting an in-person hiring drive for the position of Python Data Engineer in Bengaluru on 31st October 2026. The interview location details are mentioned below:

Persistent Systems Ltd, 4th and 5th Floor, Block 9, Pritech Park - SEZ, Behind RMZ Ecospace, Survey Nos. 51 to 64/4 & 66/1, Bellandur Village, Varthur Hobli, Bengaluru, Karnataka - 560103.

We are looking for an experienced Python / Spark Data Engineer with strong expertise in building scalable data processing and integration solutions on modern cloud data platforms. The ideal candidate will have hands-on experience with Python, Apache Spark/PySpark, SQL, Airflow, and CI/CD technologies, along with a strong understanding of data engineering best practices. You will be responsible for developing high-performance data pipelines, enabling large-scale data processing, and supporting enterprise analytics initiatives across cloud and distributed environments.

• Experience: 5 to 10 Years
• Job Type: Full-Time Employment

What You'll Do:

• Design, develop, and maintain scalable Python and PySpark-based data pipelines for batch and large-scale data processing.
• Build reusable frameworks and components for data ingestion, transformation, validation, enrichment, and publishing.
• Process structured and semi-structured datasets from databases, APIs, cloud platforms, files, and enterprise applications.
• Develop, optimize, and maintain complex SQL queries and Spark transformations for high-volume datasets.
• Implement robust data-quality frameworks, reconciliation checks, exception handling, monitoring, and logging mechanisms.
• Develop modular, maintainable, testable, and reusable Python/PySpark code following engineering best practices.
• Manage source code repositories using GitHub, including branching, pull requests, code reviews, and release workflows.
• Build and maintain CI/CD pipelines using GitHub Actions, Harness, or similar automation platforms.
• Design and manage workflow orchestration using Apache Airflow and enterprise scheduling solutions.
• Optimize Spark jobs for performance, scalability, reliability, and cost efficiency.
• Troubleshoot data pipeline failures, ETL issues, data-quality problems, and production incidents.
• Participate in code reviews, testing activities, deployment processes, and production support.
• Collaborate with Data Architects, Data Analysts, Platform Engineers, DevOps teams, and Product Owners to deliver business solutions.
• Contribute to data platform modernization and continuous improvement initiatives.
• Ensure adherence to security, governance, and data engineering standards.

Expertise You'll Bring:

• 5-10 years of experience in Data Engineering, ETL Development, or Big Data Engineering.
• Strong hands-on programming experience with Python.
• Extensive experience working with Apache Spark and PySpark.
• Strong SQL expertise with experience in complex query development and data transformation techniques.
• Experience designing and implementing production-grade ETL/ELT pipelines.
• Strong understanding of distributed computing concepts and Spark architecture.
• Experience tuning and optimizing Spark workloads for performance and scalability.
• Hands-on experience processing large-scale structured and semi-structured datasets.
• Experience working with cloud-based data platforms, preferably Google Cloud Platform (GCP).
• Knowledge of BigQuery, Dataproc, Apache Beam, and related GCP services is highly desirable.
• Experience developing and consuming REST APIs for data integration.
• Strong experience using Git/GitHub for version control, collaborative development, and code reviews.
• Practical experience with GitHub Actions, Harness, or similar CI/CD platforms.
• Experience implementing automated testing frameworks and deployment pipelines.
• Hands-on experience with Apache Airflow or equivalent workflow orchestration platforms.
• Strong understanding of logging, monitoring, alerting, error handling, and operational support processes.
• Experience troubleshooting production issues and maintaining highly available data pipelines.
• Familiarity with software engineering best practices, design patterns, and clean coding principles.
• Competitive salary and benefits package
• Culture focused on talent development with quarterly growth opportunities and company-sponsored higher education and certifications
• Opportunity to work with cutting-edge technologies
• Employee engagement initiatives such as project parties, flexible work hours, and Long Service awards
• Insurance coverage: group term life, personal accident, and Mediclaim hospitalization for self, spouse, two children, and parents

Values-Driven, People-Centric & Inclusive Work Environment:

Persistent is dedicated to fostering diversity and inclusion in the workplace. We invite applications from all qualified individuals, including those with disabilities, and regardless of gender or gender preference. We welcome diverse candidates from all backgrounds.

• We support hybrid work and flexible hours to fit diverse lifestyles.
• Our office is accessibility-friendly, with ergonomic setups and assistive technologies to support employees with physical disabilities.
• If you are a person with disabilities and have specific requirements, please inform us during the application process or at any time during your employment

"Persistent is an Equal Opportunity Employer and prohibits discrimination and harassment of any kind."