Senior Data Engineer - Bangalore - Immediate Joiners only

Infiniti Research; Bengaluru; On-site; INR 3,000,000 - 5,400,000; Full time

Job summary

Infiniti Research in Bengaluru is seeking a Senior Data Engineer with 5+ years of experience to design, build, and optimize enterprise-scale ETL pipelines using Databricks and AWS Cloud. The role focuses on transforming complex datasets into actionable analytics for spend, patient access, and commercial reporting initiatives.

You will develop scalable Databricks jobs (PySpark/SQL), manage data ingestion and validation, and collaborate with BI teams to deliver reliable data for dashboards.

Qualifications

• 5+ years of experience in data engineering with Databricks.
• Strong hands-on with PySpark and SQL in Databricks environments.
• Experience building and optimizing ETL pipelines for large datasets.

Responsibilities

• Design, develop, and maintain scalable ETL pipelines in Databricks (PySpark/SQL).
• Transform AWS assets into Databricks workflows and ensure data quality for analytics.
• Deploy data pipelines with monitoring, validation, and documentation.

Skills

PySpark

SQL

Tools

Databricks

AWS Cloud

Airflow

Unity Catalog

Job description

Job description

Senior Data Engineer

Strictly Immediate Joiners only(0 to 15 days NP)

Exp : Minimum 5+ Years

Skills : AWS , Databricks

Job Description:

We are seeking a highly skilled Data Engineer with expertise in Databricks and AWS Cloud to design, build, and optimize enterprise-scale ETL pipelines and reporting solutions. This role is ideal for someone who thrives at the intersection of data engineering and business intelligence, with a strong focus on transforming complex datasets into inputs for actionable insights. You will play a critical role in enabling spend analytics, patient access, and commercial reporting initiatives.

Key Responsibilities Databricks ETL Development

• Design, develop, and maintain scalable ETL pipelines in Databricks using PySpark and SQL.
• Implement robust data ingestion, transformation, and validation processes to ensure high-quality datasets for analytics.
• Optimize workflows for performance, scalability, and reliability across large healthcare datasets.

Data Product Build

• Transform AWS assets (on EC2 and Redshift) to Databricks by recreating the running ETL jobs and orchestrating via Airflow or Databricks Workflows.
• Hands-on working experience of catalog management using Unity Catalog.
• Conversant with AI capabilities within AWS and Databricks.

Requirements

• 7+ years of experience in Databricks data engineering roles.
• Strong hands‑on proficiency in Databricks (PySpark, SQL) and AWS Cloud.
• Proven track record in ETL pipeline development and understanding of BI dashboards.
• Experience with US healthcare datasets.
• Strong SQL skills for data extraction, aggregation, and reporting.
• Excellent problem-solving abilities, with the capacity to work independently and in cross‑functional teams.
• Detail-oriented with a commitment to delivering high-quality, reliable data solutions.

Good to Have

Domain Data Knowledge

• Knowledge of IQVIA claims (standard, eLaaD, Remit, Rejection, NBRx, TRx, IGG4 Claims) and MMIT datasets to derive actionable insights.

End-to-End Production Deployment Orchestration

Candidates with exposure to deploying and orchestrating data pipelines in production environments will be preferred. While not a core requirement, the following experience is a strong differentiator:

• Experience deploying ETL or ML pipelines end-to-end in a production environment, including environment promotion across dev, staging, and production.
• Familiarity with CI/CD tooling (GitHub Actions, Azure DevOps, or Jenkins) for automating pipeline deployment and release management.
• Exposure to Databricks Asset Bundles (DABs) or equivalent frameworks for version‑controlled, repeatable job deployments.
• Working knowledge of Apache Airflow for DAG authoring, scheduling, dependency management, and monitoring of production workflows.
• Awareness of infrastructure-as-code practices (Terraform or AWS CloudFormation) for managing cloud resources supporting data pipelines.
• Basic understanding of containerization concepts (Docker) for packaging and deploying data pipeline components.
• Experience with pipeline monitoring and alerting tracking job health, SLA adherence, failure notifications, and data freshness in production.
• Familiarity with secrets and configuration management across environments (AWS Secrets Manager, Databricks Secrets, or equivalent).