Data Engineer (Databricks)

Affine Analytics; Gurugram District, Bengaluru, Hyderabad; Hybrid; INR 1,000,000 - 1,500,000; Full time

Job summary

Affine Analytics is looking for a skilled Data Engineer to design, build, and maintain scalable data solutions on the Azure platform. The role involves developing robust data pipelines, ensuring data quality, and contributing to enterprise-wide analytics initiatives.

The ideal candidate will collaborate with various teams, monitor data pipelines, and lead data engineering projects, applying effective change management practices.

Qualifications

• Strong experience with Azure ecosystem including Azure Databricks, Data Factory, and ADLS.
• Proficiency in PySpark and Spark SQL.
• Solid understanding of ETL processes and data pipeline development.
• Advanced SQL skills.

Responsibilities

• Design and develop scalable data pipelines using Spark SQL and PySpark.
• Build and orchestrate ETL workflows using Azure Data Factory.
• Develop and maintain a modern Lakehouse architecture with ADLS and Databricks.
• Perform data preparation tasks including cleaning and normalization.
• Collaborate with DevOps teams for production deployments.

Skills

Azure ecosystem

PySpark

Spark SQL

ETL processes

Advanced SQL

Job description

Job Description

We are seeking a skilled Data Engineer to design, build, and maintain scalable data solutions on the Azure platform. The ideal candidate will play a key role in developing robust data pipelines, ensuring data quality, and contributing to enterprise-wide analytics initiatives.

Key Responsibilities

• Design and develop scalable data pipelines using Spark SQL and PySpark on Azure Databricks
• Build and orchestrate ETL workflows using Azure Data Factory (ADF)
• Develop and maintain a modern Lakehouse architecture leveraging ADLS and Databricks
• Perform data preparation tasks including data cleaning, normalization, deduplication, and data type transformations
• Collaborate with DevOps teams to deploy and manage solutions in production environments
• Monitor data pipelines, identify issues, and implement corrective actions, including root cause analysis and resolution
• Actively contribute as part of the global Analytics team, delivering data-driven insights and solutions
• Collaborate with Data Science and Business Intelligence teams to share best practices and drive innovation
• Lead data engineering projects and support cross-functional initiatives led by other team members
• Apply change management practices, including documentation, communication, and training, to support system upgrades and data migrations

Must-Have Skills

• Strong experience with Azure ecosystem (Azure Databricks, Azure Data Factory, ADLS)
• Proficiency in PySpark and Spark SQL
• Solid understanding of ETL processes and data pipeline development
• Advanced SQL skills