Data Engineer

Ambiquest Hyderabad; Dadri, Hyderabad, Bengaluru; Ibrido; INR 2.800.000 - 4.600.000; Tempo pieno

Descrizione del lavoro

Ambiquest Hyderabad is seeking an experienced Senior Data Engineer with hands-on expertise in Azure Databricks, Spark, Scala, and SQL to design, develop, and optimize enterprise-grade data engineering solutions on Azure.

The role involves building scalable batch and streaming data pipelines, implementing ETL/ELT, optimizing Spark workloads, and supporting QA/UAT and production deployment. Healthcare domain exposure is a plus.

Competenze

• 5+ years hands-on data engineering / data warehousing experience.
• Experience designing scalable enterprise data pipelines.
• Strong knowledge of data integration, transformation, and processing frameworks.
• Experience in Agile development environments.

Mansioni

• Design, develop, and implement scalable data engineering solutions using Azure Databricks and ADLS.
• Build and maintain ETL/ELT pipelines for batch and streaming data processing.
• Develop data transformation workflows using Spark, Scala, and SQL.
• Implement streaming solutions with Spark Structured Streaming and patterns.
• Optimize Spark applications and SQL queries for performance and reliability.
• Lead and mentor data engineering teams and participate in code reviews.
• Collaborate with architects, analysts, and cross-functional teams in Agile delivery.

Conoscenze

Azure Databricks

ADLS

Apache Spark

Scala

SQL

Data Streaming

Healthcare domain

Descrizione del lavoro

Dear Candidates,

We have an urgent opening for Azure Data Engineer role for one of our reputed clients.

Job Title: Senior Data Engineer Azure Databricks & SQL

Experience: 5+ Years
Primary Skills: Azure Databricks, ADLS, Apache Spark, Scala, SQL, Data Streaming
Preferred Domain: Healthcare

Job Summary

We are looking for an experienced Senior Data Engineer with strong hands‑on expertise in Azure Databricks, Apache Spark, Scala, and SQL to design, develop, and optimize enterprise‑grade data engineering solutions on Azure.

The ideal candidate should have experience building scalable batch and streaming data pipelines, implementing ETL/ELT solutions, optimizing Spark workloads, and working with Azure Data Lake Storage (ADLS). The candidate will also provide technical leadership, mentor team members, and support QA/UAT and production implementation activities.

Key Responsibilities

• Design, develop, and implement scalable data engineering solutions using Azure Databricks and ADLS.
• Build and maintain enterprise‑grade ETL/ELT pipelines for batch and streaming data processing.
• Develop data transformation workflows using Apache Spark, Scala, and SQL.
• Implement streaming solutions using Spark Structured Streaming and relevant streaming design patterns.
• Optimize Spark applications and SQL queries to improve performance, scalability, and reliability.
• Design reusable and standardized components for data ingestion, transformation, and integration.
• Develop and maintain data pipelines across different data zones, including Raw, Curated, and Consumption layers.
• Implement data quality frameworks, validation rules, and data governance practices.
• Identify opportunities to improve existing solutions through automation, performance tuning, and operational efficiency.
• Lead and mentor data engineering teams by providing technical guidance, conducting code reviews, and supporting solution design.
• Collaborate with architects, business analysts, and cross‑functional teams in Agile delivery environments.
• Support QA, UAT, deployment, production implementation, and troubleshooting activities.
• Ensure adherence to engineering standards, documentation, logging, monitoring, and CI/CD practices.

Mandatory Technical Skills

• Strong hands‑on experience with Azure Databricks.
• Experience with Azure Data Lake Storage (ADLS).
• Strong knowledge of Apache Spark, including core concepts and performance tuning.
• Proficiency in Scala programming.
• Strong experience in SQL, including query development and performance optimization.
• Hands‑on experience with Data Streaming, particularly Spark Structured Streaming and streaming design patterns.
• Experience designing and developing enterprise‑grade ETL/ELT data pipelines.

Good-to-Have Skills

• Experience with Python and PySpark.
• General knowledge of the Azure cloud platform.
• Experience working with MongoDB.
• Healthcare domain experience, preferably involving:
  • Healthcare Claims
  • Clinical Data
  • Eligibility
  • Provider Data
  • Member Data
  • HIPAA‑aware data handling

Required Experience & Competencies

• Minimum 5+ years of hands‑on experience in Data Engineering / Data Warehousing development and operations.
• Proven experience designing and implementing scalable enterprise data pipelines.
• Strong understanding of data integration, transformation, and processing frameworks.
• Experience working in Agile development environments.
• Ability to lead technical discussions and provide guidance to engineering teams.
• Strong analytical, problem‑solving, and troubleshooting skills.
• Excellent communication and collaboration skills.