Data Engineer-Information Technology

Emcure Pharmaceuticals Limited; Pune District; On-site; INR 1,400,000 - 2,000,000; Full time

Job summary

Emcure Pharmaceuticals Limited is seeking a data engineer to design, develop, and maintain data pipelines using Databricks and Azure Data Factory. The role focuses on integrating diverse data sources, ensuring data quality, and building scalable data transformations to support BI and DW solutions.

The ideal candidate will have strong SQL, Python, and Spark skills, with experience in data warehousing and ETL principles, and will collaborate with business teams to translate requirements into

Qualifications

• Strong experience with Databricks notebooks and Spark in data processing and transformations.
• Proficiency in SQL, Python, and Spark; ability to design scalable data pipelines.
• Familiarity with ETL principles, data governance, and data quality frameworks.

Responsibilities

• Design, develop, and maintain data pipelines and ETL processes using Databricks and ADF.
• Integrate data from multiple sources and ensure data quality and accuracy.
• Build reusable data quality rules and optimize pipelines for performance and scalability.
• Act as liaison between data analysts, data scientists, and business teams; document resources.
• Ensure compliance with data governance and other legal requirements.

Skills

SQL

Python

Spark

Databricks

Azure Data Factory

ETL

Data Quality

Data Warehousing

Big Data

BRD (Business Requirements Document)

BI & DW

Qlik Sense

MS Power BI

Education

Bachelor's degree in computer science or related field

Tools

Databricks

Azure Data Factory

SSIS

Qlik Sense

MS Power BI

Job description

Responsible for designing, developing, and maintaining data pipelines and ETL processes using Databricks and Azure Data Factory.

• Good experience in using SQL, Python and Spark within Databricks notebooks for data processing and transformations.
• Integrate and orchestrate data from various sources, ensuring data quality and accuracy.
• Build reusable rules for Data Quality and required transformations.
• Optimize, monitor data pipelines for performance and scalability and debug in case of issues.
• Proficiency in Databricks, Azure Data Factory, Python, SQL is must.
• Familiarity with ETL principles, data warehousing, and big data technologies.
• Evaluate and define technical specs using BRD for BI and DW solutions.
• Build conceptual and logical models based on the functional flow of business in scalable way.
• Work with Business functions/ leadership and Application SMEs to understand the requirement and analysing the source to fulfil the requirements.
• Be compliant to data quality, governance, compliance, and other legal requirements.
• Act as key P.O.C between data analyst, data scientists, and the business/application teams.
• Must have Azure Databricks experience & Data Integrations.
• Good work experience on Data bricks, ADF, and other Azure and Databricks objects.
• Experience on creating frameworks towards building the optimized data pipelines is a must.
• Working knowledge of different self-hosted and other runtimes along with SSIS.
• Strong work experience of SQL, Scala, Python and Spark.
• Performance Tuning of Spark SQL running on DELTA LAKE storage and Strong Knowledge on Databricks and Cluster Configurations.
• Working knowledge of Data Warehousing, Modelling, Data Validation, Data Migration.
• Documentation of all project resources and follow best practices for development.
• Knowledge of front-end tools like Qlik Sense, MS PBI
• Minimum bachelors degree in computer science or related field is required.

Strong problem-solving skills, good communication abilities and go-getter attitude.