Data Engineer

Quantiphi Analytics Solutions; Navi Mumbai; On-site; INR 1,800,000 - 3,000,000; Full time

Job summary

Quantiphi Analytics Solutions is seeking a Data Engineer to design and deliver big data pipelines in the healthcare domain from its Airoli, Mumbai location. You will work with Azure data services to ingest, transform, and govern large structured and unstructured data across geographies.

The role emphasizes Databricks, Unity Catalog, Delta Lake, and scalable ETL/ELT pipelines using PySpark. Collaboration with cloud, software, and ML teams is essential.

Qualifications

• 3+ years hands-on data engineering with PySpark and Python.
• Deep Databricks expertise including Unity Catalog, Delta Lake, and Workflows.
• Strong SQL skills for data transformation, aggregation, and tuning.
• Knowledge of dimensional modeling (Star/Snowflake) and SCDs.
• Experience with Azure data services: ADF, ADLS Gen2, and Logic Apps.
• Familiarity with data warehousing concepts and distributed systems.

Responsibilities

• Develop and maintain big data pipelines for healthcare data.
• Collaborate with cloud, software, and ML engineers to build platforms.
• Design data models and warehousing solutions on Azure Synapse / Databricks.
• Implement governance, lineage, and secure access using Unity Catalog.
• Build scalable ETL/ELT using PySpark on Databricks and manage scheduling.
• Automate workflows and ensure data quality across terabytes to petabytes daily.

Skills

PySpark

Python

Databricks

Unity Catalog

Delta Lake

SQL

Data Modeling

Azure Data Factory

ADLS Gen 2

APIs & Integration

Tools

Azure Databricks

Databricks Workflows

Azure Synapse

Python tooling

Job description

Role: Data Engineer

Experience Level: 2 to 4 Years

Work location: Airoli, Mumbai

What youll do:

You will be working as a Data Engineer within the healthcare domain, designing and delivering big data pipelines for structured and unstructured data that are running across multiple geographies, helping healthcare organizations achieve their business goals with use of data ingestion technologies, cloud services & DevOps. You will be working with Architects from other specialties such as Cloud engineering, Software engineering, ML engineering to create platforms, solutions and applications that cater to latest trends in the healthcare industry such as data intelligence platforms, enabling advanced analytics across pharmaceutical research, drug commercialization, sales & marketing, supply chain, and other key business domains, amongst others.

Role & Responsibilities:

• Work with cloud engineers and customers to solve complex big data problems by developing utilities for migration, storage, and processing on Azure Cloud.
• Design and build a cloud migration strategy for cloud and on-premise applications, leveraging Azure-native services.
• Develop, schedule, and manage data pipelines using Azure Data Factory (ADF) and Databricks Workflows.
• Design and implement data models following best practices including Star Schema, Snowflake Schema, and Slowly Changing Dimensions (SCDs – Type 1, 2, 3).
• Build and maintain data warehousing solutions on Azure Synapse Analytics / Databricks SQL Warehouse, ensuring data quality, consistency, and performance.
• Develop scalable ETL/ELT pipelines using PySpark on Databricks, processing terabytes to petabytes of data daily.
• Implement and manage Unity Catalog for fine-grained data governance, access control, lineage tracking, and data discovery across Databricks workspaces.
• Leverage Azure Data Lake Storage Gen 2 (ADLS Gen 2) for scalable, secure, and hierarchical data storage.
• Automate workflows and integrate systems using Azure Logic Apps for event-driven orchestration and business process automation.
• Diagnose and troubleshoot complex distributed systems problems and develop solutions with significant impact at massive scale.
• Communicate with a wide set of teams including Infrastructure, Network, Engineering, DevOps, and cloud customers.
• Build advanced tooling for automation, testing, monitoring, administration, and data operations across multiple cloud clusters.

Skills expectation:

• Must have:
  • 3+ years of hands-on experience in data engineering with strong proficiency in PySpark and Python.
  • Deep expertise in Databricks, including:
  • Unity Catalog – data governance, access control, data lineage, and cataloging.
  • Delta Lake – ACID transactions, time travel, and optimized storage.
  • Databricks Workflows – pipeline orchestration and scheduling.
  • Strong SQL and Advanced SQL skills for data transformation, aggregation, and performance tuning.
  • Solid understanding of Data Modelling concepts:
  • Dimensional modelling (Star & Snowflake schemas)
  • Slowly Changing Dimensions (SCDs) – Type 1, 2, and 3
  • Data Vault and Medallion Architecture (Bronze/Silver/Gold layers)
  • Hands‑on experience with Data Warehousing principles and implementation.
  • Proficiency in Azure Services:
  • Azure Data Factory (ADF) – pipeline creation, triggers, linked services, and monitoring.
  • Azure Data Lake Storage Gen 2 (ADLS Gen 2) – hierarchical namespace, access control, and integration.
  • Azure Logic Apps – workflow automation and system integration.
  • Experience building and supporting large-scale data systems in production environments.
  • Familiarity with NoSQL databases and cloud-native data services.
  • Strong understanding of distributed systems and data structures.
• Good to have:
  • Experience working with healthcare and pharmaceutical Data
  • Designing and development of ETL pipeline
  • Requirement gathering and understanding of the problem statement
  • End-to-end ownership of the entire delivery of the project
  • Designing and documentation of the solution
  • Team management and mentoring

Leadership qualities

• Ability to lead technology teams and provide them mentorship / support to accelerate performance.
• Experience in leading multiple large projects as well as a deep understanding of Agile developments
• Effective communication with all the stakeholders involved.
• Communicate clearly about complex subjects and technical plans with technical and non technical audiences.

Behavioural skills:

• Effective communication with all the stakeholders involved
• Need to communicate clearly about complex subjects and technical plans
• Ability to mentor and groom junior members of the team and provide them with guidance and roadmap.
• Must also have the ability to interact with other members of the team (Juniors, ATA, TA, BA, etc) to get their designs from concept to development

Keeping various audiences in mind, engineers must write their reports in clear language accessible to all.

What is in it for you:

• Opportunity to learn cloud native services and how to utilize those services to solve various business problems.
• More hands‑on learning opportunity in Databricks, Python, SQL, etc
• Sponsored certification opportunity for various courses of your choice (eg, GCP, AWS, Azure, Tableau, Looker,etc)
• Allocated work will allow you to not only just complete the task but you will be given an opportunity to own-up the work and take responsibility of it’s end-to-end delivery