Data Engineer - Pharma Commercial Data

Patch Infotech; Hyderabad; On-site; INR 1,200,000 - 2,400,000; Full time

Job summary

Patch Infotech is seeking a skilled Data Engineer to design, build, and optimize scalable data pipelines using SQL, Python, PySpark, and Azure Databricks. You will work with pharmaceutical commercial datasets (claims, Rx, patient, sales, payer) to support analytics and BI initiatives.

The role focuses on ETL/ELT development, data ingestion from diverse sources, data quality checks, and collaboration with analysts and data scientists to deliver reliable datasets, with emphasis on governance and

Qualifications

• Strong SQL programming and query optimisation skills.
• Proficiency in Python for data engineering and automation.
• Hands-on experience with PySpark and Apache Spark.
• Experience developing ETL/ELT pipelines.
• Strong experience with Azure Databricks.
• Experience with Azure Data Lake Storage (ADLS).
• Knowledge of Azure Data Factory (ADF) for orchestration.
• Familiarity with Delta Lake, Parquet, and other big data storage formats.
• Experience with Git or Azure DevOps for source control.
• Good understanding of data modelling and data warehousing concepts.
• Strong analytical and problem-solving skills.
• Bachelor's or Master's degree in Computer Science, Information Technology, Engineering, or a related field.

Responsibilities

• Design, develop, and maintain scalable ETL/ELT pipelines using Python, PySpark, and SQL.
• Build and optimise data processing workflows in Azure Databricks.
• Develop and maintain data ingestion frameworks from multiple structured and unstructured data sources.
• Process and transform pharmaceutical commercial datasets, including claims, Rx, payer, sales, formulary, and patient-level data.
• Optimise Spark jobs for performance, scalability, and cost efficiency.
• Implement data quality checks, validation rules, and monitoring processes.
• Collaborate with business analysts, data scientists, and commercial analytics teams to deliver reliable datasets.
• Develop reusable data engineering components and automation frameworks.
• Troubleshoot data pipeline issues and ensure high availability of data platforms.
• Participate in code reviews and follow best practices for version control and CI/CD.
• Ensure compliance with healthcare data governance, privacy, and security standards.

Skills

SQL

Python

PySpark

ETL development

Azure Databricks

ADLS

ADF

Delta Lake

Parquet

Git/Azure DevOps

Data warehousing

Data modelling

Analytics

CI/CD

Spark tuning

Education

Bachelor's or Master's degree in Computer Science, Information Technology, Engineering, or a related field

Tools

Azure Data Factory (ADF)

Azure Data Lake Storage (ADLS)

Azure Databricks

Delta Live Tables

Databricks Workflows

Synapse Analytics

Microsoft Fabric

Job description

We are seeking a skilled Data Engineer with strong expertise in SQL, Python, PySpark, ETL development, and Azure Databricks to design, build, and optimise scalable data pipelines. The ideal candidate should have hands- on experience working with pharmaceutical commercial datasets, including **claims, prescription (Rx), patient, sales, and payer data, and be proficient in processing large-scale healthcare data to support analytics and business intelligence initiatives.

Responsibilities:

• Design, develop, and maintain scalable ETL/ELT pipelines using Python, PySpark, and SQL.
• Build and optimise data processing workflows in Azure Databricks.
• Develop and maintain data ingestion frameworks from multiple structured and unstructured data sources.
• Process and transform pharmaceutical commercial datasets, including claims, prescription, payer, sales, formulary, and patient-level data.
• Optimise Spark jobs for performance, scalability, and cost efficiency.
• Implement data quality checks, validation rules, and monitoring processes.
• Collaborate with business analysts, data scientists, and commercial analytics teams to deliver reliable datasets.
• Develop reusable data engineering components and automation frameworks.
• Troubleshoot data pipeline issues and ensure high availability of data platforms.
• Participate in code reviews and follow best practices for version control and CI/CD.
• Ensure compliance with healthcare data governance, privacy, and security standards.

Requirements:

• Strong SQL programming and query optimisation skills.
• Proficiency in Python for data engineering and automation.
• Hands- on experience with PySpark and Apache Spark.
• Experience developing ETL/ELT pipelines.
• Strong experience with Azure Databricks.
• Experience with Azure Data Lake Storage (ADLS).
• Knowledge of Azure Data Factory (ADF) for orchestration.
• Familiarity with Delta Lake, Parquet, and other big data storage formats.
• Experience with Git or Azure DevOps for source control.
• Good understanding of data modelling and data warehousing concepts.
• Strong analytical and problem- solving skills.
• Bachelor's or Master's degree in Computer Science, Information Technology, Engineering, or a related field.

Domain Expertise:

• Experience working with pharmaceutical commercial datasets is mandatory.
• Hands- on knowledge of one or more of the following: Medical Claims Data, Pharmacy Claims Data, Prescription (Rx) Data, Patient- Level Data, Sales and Commercial Data, Payer and Formulary Data.
• Understanding of healthcare data standards and commercial analytics workflows is highly preferred.

Preferred Qualifications:

• Experience with Azure Data Factory, Synapse Analytics, or Microsoft Fabric.
• Knowledge of Delta Live Tables and Databricks Workflows.
• Familiarity with data governance and data catalogue tools.
• Experience with performance tuning for Spark workloads.
• Exposure to Agile/Scrum development methodologies.
• Healthcare or Life Sciences industry experience.

Preferred Certifications:

• Microsoft Certified: Azure Data Engineer Associate.
• Databricks Certified Data Engineer Associate/Professional.

Key Competencies:

• Strong analytical and troubleshooting skills.
• Excellent communication and stakeholder management.
• Ability to work independently and collaboratively.
• Attention to detail and commitment to data quality.
• Continuous learning mindset and adaptability.