Area Manager-Data Engineering-Cloud Data Engineering

EXL; Dadri; On-site; INR 1,500,000 - 2,200,000; Full time

Job summary

EXL is seeking an Azure Data Engineer in the Healthcare Analytics BU to design, build, and optimize data pipelines using Azure Databricks, PySpark, and Spark SQL. You will implement data orchestration with Azure Data Factory and Databricks Workflows to automate ETL/ELT processes.

You will also establish CI/CD pipelines for notebooks and scripts, develop healthcare data models, and contribute to enterprise BI architectures and dashboards for data-driven decision making.

Qualifications

• MUST have 6+ years of experience as a Data Engineer with cloud data pipelines.
• MUST have 3+ years of hands-on experience with Azure Databricks.
• Strong experience with core Azure data services – ADLS Gen2, Azure Data Factory.
• Proficiency in Python and expert-level skills in SQL.
• Solid understanding of data warehousing concepts and dimensional modeling.
• Experience with CI/CD pipelines (Azure DevOps, Jenkins, GitHub Actions) and version control (Git).

Responsibilities

• Design, develop, test, and deploy robust and scalable data pipelines using Azure Databricks, PySpark, and Spark SQL for processing large volumes of structured and unstructured data.
• Implement workflow orchestration using Azure Data Factory and Databricks Workflows to automate ETL/ELT processes.
• Implement and maintain CI/CD pipelines for Databricks notebooks, scripts, and configurations using Azure DevOps or GitHub Actions, promoting code across dev, test, and prod environments.
• Develop and maintain healthcare data models, including data entities, relationships, and data flows, to support healthcare analytics and decision-making.
• Design and implement enterprise-wide BI architectures. Develop and maintain data warehouses and data marts.
• Develop and maintain reports, dashboards, and data visualizations. Collaborate with stakeholders to understand business needs and provide data-driven insights

Skills

Python

SQL

Data warehousing concepts

Project management

Stakeholder communication

Strategic problem solving

Tools

Azure Databricks

PySpark

Spark SQL

Azure Data Factory

CI/CD pipelines

Azure DevOps

GitHub Actions

Job description

• Design, develop, test, and deploy robust and scalable data pipelines using Azure Databricks, PySpark, and Spark SQL for processing large volumes of structured and unstructured data.
• Implement workflow orchestration using Azure Data Factory and Databricks Workflows to automate ETL/ELT processes.
• Implement and maintain CI/CD pipelines for Databricks notebooks, scripts, and configurations using Azure DevOps or GitHub Actions, promoting code across dev, test, and prod environments.
• Develop and maintain healthcare data models, including data entities, relationships, and data flows, to support healthcare analytics and decision-making.
• Design and implement enterprise-wide BI architectures. Develop and maintain data warehouses and data marts.
• Develop and maintain reports, dashboards, and data visualizations. Collaborate with stakeholders to understand business needs and provide data-driven insights

Role Title: Azure Data Engineer

BU/Segment: Healthcare Analytics

Responsibilities

• Design, develop, test, and deploy robust and scalable data pipelines using Azure Databricks, PySpark, and Spark SQL for processing large volumes of structured and unstructured data.
• Implement workflow orchestration using Azure Data Factory and Databricks Workflows to automate ETL/ELT processes.
• Implement and maintain CI/CD pipelines for Databricks notebooks, scripts, and configurations using Azure DevOps or GitHub Actions, promoting code across dev, test, and prod environments.
• Develop and maintain healthcare data models, including data entities, relationships, and data flows, to support healthcare analytics and decision-making.
• Design and implement enterprise-wide BI architectures. Develop and maintain data warehouses and data marts.
• Develop and maintain reports, dashboards, and data visualizations. Collaborate with stakeholders to understand business needs and provide data-driven insights

Person attributes

• 6+ years of experience as a Data Engineer with a proven track record of building data pipelines in the cloud.
• 3+ years of hands-on experience with Azure Databricks
• Strong experience with core Azure data services – Azure Data Lake Storage (Gen 2), Azure data factory.
• Proficiency in Python and expert-level skills in SQL
• Solid understanding of data warehousing concepts, dimensional modeling (star/snowflake schemas), and data architecture principles
• Experience with CI/CD pipelines (Azure DevOps, Jenkins, GitHub Actions) and version control systems (Git).
• Demonstrated experience and knowledge of strategic problem solving and frameworks, and project management skills
• Excellent written and verbal communication with the ability to establish credibility and strong relationships with stakeholders