Senior Data Engineer- Azure

RAF Solutions Llp; Ahmedabad District; On-site; INR 1,500,000 - 2,100,000; Full time

Job summary

RAF Solutions LLP is seeking a Senior Data Engineer to drive scalable data pipelines for a migration to a modern Azure Native architecture. The role focuses on building an AI-ready data platform using Fabric IQ and facilitating AI/ML workflows through Foundry IQ.

Key work includes designing robust pipelines, migrating existing systems to Fabric-native architecture, and implementing the Medallion Architecture for healthcare data.

Qualifications

• 4–6 years of experience in data engineering or equivalent.
• Strong hands-on experience with Azure Data Platform.
• Proven track record building and optimizing large-scale data pipelines.
• Experience in data platform modernization or migration projects.

Responsibilities

• Design and build robust, scalable data pipelines (batch + near real-time).
• Lead migration of existing pipelines to Fabric-native architecture.
• Implement Medallion Architecture (Bronze/Silver/Gold) for healthcare data.
• Develop data models and transformations for analytics and AI use cases.
• Ensure data quality, validation, and observability across pipelines.
• Enable AI/ML workflows by preparing datasets for RAG pipelines and model consumption.
• Collaborate with Architects, AI/ML engineers, and Product teams.
• Mentor junior engineers and promote engineering best practices.

Skills

Python

PySpark

SQL

Tools

Microsoft Fabric (Fabric IQ)

Azure Data Factory

Azure Synapse

Data Lake

Job description

Role Summary

We are hiring a Senior Data Engineer to drive the development of scalable, high-performance data pipelines for a migration project to a modern Azure Native architecture. This role will be instrumental in building an AI-ready data platform using Microsoft Fabric (Fabric IQ) and enabling seamless integration with AI/ML workflows (Foundry IQ).

Key Responsibilities

• Design and build robust, scalable data pipelines (batch + near real-time) using Microsoft Fabric (Data Pipelines, Lakehouse)
• Lead migration of existing pipelines (AKS/custom ETL) to Fabric-native architecture
• Implement Medallion Architecture (Bronze Silver Gold) for structured and unstructured healthcare data
• Develop and optimize data models and transformations for analytics and AI use cases
• Build ingestion frameworks for:
• APIs (FHIR, EHR systems)
• Files (JSON, HL7, CSV)
• Ensure data quality, validation (FHIR/claims), and observability across pipelines
• Enable AI/ML workflows by preparing datasets for:
• RAG pipelines
• Feature engineering and model consumption
• Optimize pipeline performance, cost, and scalability
• Collaborate with Architects, AI/ML engineers, and Product teams to translate requirements into data solutions
• Mentor junior engineers and drive engineering best practices

Required Experience

• 4-6 years of experience (3+ years in data engineering)
• Strong hands-on experience with Azure Data Platform
• Proven experience in building and optimizing large-scale data pipelines
• Experience in data platform modernization or migration projects

Technical Skills

• Languages: Python, PySpark, SQL (advanced)
• Azure / Fabric:
  • Microsoft Fabric (preferred) OR Azure Data Factory, Synapse, Data Lake
• Data Architecture:
  • Lakehouse, Medallion Architecture
  • Batch + streaming data processing
• Data Handling:
  • JSON, Parquet, CSV, API integrations
• Performance Optimization:
  • Partitioning, indexing, query tuning

Nice to Have

• Experience with healthcare data (FHIR, claims, EHR)
• Exposure to RAG pipelines / vector databases / AI data prep
• Familiarity with event-driven architectures
• Understanding of data security & governance in regulated environments

Success Metrics

• High-quality, scalable pipelines delivered on time
• Seamless migration with minimal business disruption
• Improved data performance, reliability, and cost efficiency
• Strong enablement of AI/ML and analytics use cases

What Were Looking For

• Strong hands-on engineer with ownership mindset
• Ability to operate in fast-paced, evolving architecture environment
• Problem-solver with focus on performance, scalability, and data quality
• Mentor and team player

Why This Role Matters

This role is critical to building the foundation of AI-first platform, enabling:

• Scalable data processing via Fabric IQ
• AI-driven intelligence via Foundry IQ
• Future-ready healthcare data platform