Data Engineer

Intact Insurance; Gurugram District; On-site; INR 1,200,000 - 2,400,000; Full time

Job summary

Intact Insurance is seeking a highly skilled Data Engineer to design, develop, and maintain scalable data solutions. The role demands hands-on work with distributed data processing, modern data platforms, and strong software engineering practices.

You will build and optimize ingestion, transformation, and serving pipelines using Azure Databricks, PySpark, Delta Lake, and Azure data services, while ensuring data quality and observability.

Qualifications

• Proficiency in Python/Java/Scala and SQL required.
• Experience with Spark or PySpark in distributed processing.
• Hands-on with Databricks or similar platforms.
• Ability to design scalable data pipelines and warehousing concepts.
• Experience with Git and CI/CD practices.
• Experience with data quality, observability, and monitoring frameworks.
• Strong problem-solving and debugging skills.
• Production-grade data solutions experience.

Responsibilities

• Design, develop, and maintain scalable ingestion, transformation, and serving pipelines.
• Build and optimize data solutions using Azure Databricks, PySpark, Delta Lake, and Azure data services.
• Develop data integration workflows with Azure Data Factory and other Azure-native services.
• Design data models to support analytics and downstream applications.
• Implement data quality, validation, monitoring, and observability.
• Develop automated testing strategies and maintain high code quality.
• Implement CI/CD pipelines and deployment automation using Azure DevOps.
• Collaborate with architects, product owners, analysts, and stakeholders to deliver data products.
• Troubleshoot production issues and improve platform reliability, scalability, performance.
• Ensure adherence to data governance and security best practices.

Skills

Python

Java

Scala

SQL

Spark/PySpark

Databricks

Data Pipelines

Data Warehousing

Git

Testing Frameworks

CI/CD

Data Governance

Job description

We are seeking a highly skilled Data Engineer to design, develop, and maintain scalable, secure, and high-performance data solutions. The ideal candidate will be a strong analytical problem solver with the ability to independently design, develop, test, and deliver end-to-end data engineering solutions. The role requires hands-on experience with distributed data processing technologies, modern data platforms, and software engineering best practices, including testing and CI/CD.

Key Responsibilities

• Design, develop, and maintain scalable data ingestion, transformation, and serving pipelines.
• Build and optimize data solutions using Azure Databricks, PySpark, Delta Lake, and Azure data services.
• Develop data integration workflows using Azure Data Factory and other Azure-native services.
• Design and implement data models that support analytics, reporting, and downstream applications.
• Implement robust data quality, validation, monitoring, and observability solutions.
• Develop automated testing strategies and maintain high code quality standards.
• Implement CI/CD pipelines and deployment automation using Azure DevOps.
• Collaborate with architects, product owners, analysts, and business stakeholders to deliver data products.
• Troubleshoot production issues and continuously improve platform reliability, scalability, and performance.
• Ensure adherence to data governance, security, and engineering best practices.

Mandatory Skills

• Programming skills in Python, Java, Scala, or another modern programming language.
• Advanced SQL skills.
• Hands-on experience with Apache Spark / PySpark.
• Experience with Databricks or similar distributed data processing platforms.
• Experience building scalable data pipelines and data integration solutions.
• Good understanding of data warehousing and dimensional modelling concepts.
• Proficiency with Git or any source control practices.
• Experience with automated testing frameworks such as PyTest, JUnit, or equivalent.
• Strong analytical, troubleshooting, and problem-solving skills.
• Knowledge of software engineering best practices, including code reviews, testing, and reusable design patterns.
• Experience developing and supporting production-grade data solutions.

Preferred / Good-to-Have

• Experience with Azure could.
• Experience with Databricks, Delta Lake, Delta Live Tables (DLT), or equivalent technologies.
• Experience with Azure Data Factory (ADF), Azure Synapse, AWS Glue, EMR, or comparable cloud-native data services.
• Experience with data quality, observability, and monitoring frameworks.
• Cloud or Databricks certifications.
• Experience working in Agile delivery environments.