Senior Data Engineer (PySpark)

Indigrators; Hyderabad; Remote; INR 2,400,000 - 4,200,000; Full time

Job summary

Indigrators is seeking a Senior Data Engineer (PySpark) to design, develop, and optimize large-scale data pipelines in a remote, globally collaborative setup. You will work across international teams on modern cloud-based data engineering practices.

You will contribute to ETL/ELT pipelines, ensure data quality, and document solutions while aligning with Agile/Scrum processes and project tracking tools such as Azure DevOps, Jira, and Confluence.

Qualifications

• Strong PySpark and Python skills required.
• Experience with ETL/ELT design and development.
• Experience building and maintaining data pipelines for Data Lakes.

Responsibilities

• Design, develop, and optimize ETL/ELT pipelines using PySpark and Python.
• Build scalable data flows from source systems to Data Lake environments.
• Develop and support data integration and transformation processes.
• Ensure data quality, performance, and reliability of data pipelines.
• Document technical solutions and maintain development standards.
• Collaborate with cross-functional teams in Agile Scrum environments.
• Participate in sprint planning, estimation, and delivery activities.
• Work with Azure DevOps, Jira, and Confluence for project execution.

Skills

PySpark

Python

Data Engineering

ETL/ELT

Data Lakes

SQL

Agile/Scrum

Azure DevOps/Jira

Confluence

Communication

Education

Bachelor's degree in CS/IT/Engineering

Job description

We are looking for a highly skilled Senior Data Engineer (PySpark) to join a global data transformation initiative within the pharmaceutical domain. The role involves designing, developing, and optimizing large-scale data pipelines and ETL/ELT processes on modern Data Lake platforms.

This is a fully remote opportunity with exposure to international teams, large-scale data ecosystems, and modern cloud-based data engineering practices.

Key Responsibilities

• Design, develop, and optimize ETL/ELT pipelines using PySpark and Python.
• Build scalable data flows from source systems to Data Lake environments.
• Develop and support data integration and transformation processes.
• Ensure data quality, performance, and reliability of data pipelines.
• Document technical solutions and maintain development standards.
• Collaborate with cross-functional teams in Agile Scrum environments.
• Participate in sprint planning, estimation, and delivery activities.
• Work with Azure DevOps, Jira, and Confluence for project execution.

Required Skills

• Strong experience in PySpark and Python.
• Solid background in Data Engineering.
• Hands‑on experience with ETL/ELT design and development.
• Experience working with Data Lakes and large-scale data processing.
• Strong SQL and data modeling skills.
• Experience in Agile/Scrum methodologies.
• Familiarity with Azure DevOps and/or Jira.
• Experience using Confluence for documentation.
• Excellent verbal and written communication skills.

Preferred Skills

• Experience with Palantir Foundry.
• Exposure to Databricks, Snowflake, Azure Data Factory, or similar platforms.
• Experience working with global or distributed teams.

Education

Bachelor's degree in Computer Science, Information Technology, Engineering, or a related field.