Data Engineer (Pyspark)

MantriAI; Bengaluru; On-site; INR 1,200,000 - 2,400,000; Full time

Job summary

ZAGENO is seeking a Data Engineer with PySpark expertise to own production data pipelines powering the life-sciences product catalog. You will work with large datasets using PySpark, Python, SQL, Databricks, and Delta Lake, building reliable data products for business and technical stakeholders.

The role emphasizes ownership from design through deployment, with a focus on data quality, scalability, and maintainability in a production environment.

Qualifications

• 3+ years of relevant experience in data engineering.
• Hands-on PySpark experience in production pipelines.
• Experience with Databricks and Delta Lake is preferred.

Responsibilities

• Design, develop, and maintain scalable data pipelines using PySpark, Python, and SQL.
• Own production pipelines from ingestion to deployment and monitoring.
• Troubleshoot failures and reduce fragility in data workflows and APIs.
• Improve pipeline performance and reliability through clean engineering patterns.
• Implement incremental processing and effective data partitioning strategies.
• Create reusable components to reduce duplication and debt.

Skills

Python

SQL

PySpark

Databricks

Delta Lake

Apache Airflow

CDC

Tools

Airflow

Delta Lake

Job description

Job Description: Data Engineer – PySpark
Company: ZAGENO
Role: Data Engineer – PySpark / Data Platform
Experience: 3+ years
Employment Type: Full-time
Primary Skills: Python, SQL, PySpark, Databricks, Delta Lake, Apache Airflow, Change Data Capture (CDC)
Location: Bangalore
Notice Period Preference: Candidates who can join within 30 days are preferred.

About ZAGENO

ZAGENO is a digital platform focused on simplifying the procurement of life-sciences research products. Reliable, accurate, and well-structured product data is essential to helping researchers and life-sciences organizations discover and procure the products they need.

We are looking for a hands-on Data Engineer with strong PySpark expertise and proven production pipeline ownership to strengthen the reliability, scalability, and correctness of our data platform.

In this role, you will work on the data pipelines and systems that power our life-sciences product catalog. You will address existing pipeline failures and technical debt, build efficient data synchronisation mechanisms, improve product data quality, and develop maintainable data products for business and technical stakeholders.

This is an engineering role for someone who enjoys solving real production problems, taking ownership from design through deployment, and building reliable systems that scale.

Role Overview

As a Data Engineer, you will own the design, development, testing, deployment, and operational reliability of production-grade data pipelines.

You will work with large and complex datasets using PySpark, Python, SQL, Databricks, Delta Lake, and Apache Airflow. You will also build and maintain change data capture (CDC) workflows, low-latency APIs, and data synchronisation processes.

A key part of the role involves transforming inconsistent product and supplier data into standardised, accurate, and reusable datasets. You will collaborate with Catalog Operations, Analytics, Data Science, and other business stakeholders to translate evolving requirements into robust engineering solutions.

We are looking for an engineer who can independently investigate failures, identify causes, implement lasting fixes, and continuously improve data quality and system performance.

Key Responsibilities

• Design, develop, maintain, and optimize scalable data pipelines using PySpark, Python, and SQL.
  

• Take primary ownership of production pipelines, from data ingestion and transformation through validation, deployment, and ongoing support.
  

• Troubleshoot unreliable pipelines, resolve recurring failures, and eliminate fragile implementations and manual workarounds.
  

• Improve pipeline performance, execution efficiency, reliability, and maintainability.
  

• Implement incremental processing, efficient transformations, and appropriate data partitioning strategies.
  

• Develop reusable components and engineering patterns that reduce duplication and technical debt.
  

2. Databricks and Delta Lake

• Build and maintain data processing workflows on Data bricks using PySpark.
  

• Work with Delta Lake tables and implement reliable data storage, transformation, and update patterns.
  

• Apply appropriate incremental processing and table optimization techniques.
  

• Maintain data consistency and correctness across pipeline runs