Antino - Senior Data Engineer - Databricks/Azure

Antino; Gurugram District; On-site; INR 2,400,000 - 4,000,000; Full time

Job summary

Antino in India seeks a Senior Data Engineer to lead cloud data modernization and build AI-ready data platforms. You will own scalable pipelines using Databricks, PySpark, ADF, Logic Apps, and Airflow, focusing on end-to-end orchestration and reliability.

The role requires 7+ years in data engineering, strong Python and SQL, and hands-on experience with Azure services and enterprise data architectures. Join a hands-on team driving ML/GenAI readiness and robust data governance.

Qualifications

• 7+ years of data engineering experience.
• Proficiency in Databricks (PySpark, Delta Lake), ADF, Logic Apps, and Airflow.
• Exposure to AI/ML data pipelines and MLflow.
• Strong Python and SQL programming skills.
• Experience building orchestrated data platforms with multi-tool integration.
• Understanding of ETL/ELT patterns and orchestration frameworks.
• Experience with Azure-based cloud-native data architectures.
• Hands-on with monitoring, logging and pipeline reliability.

Responsibilities

• Design, build, and maintain scalable data pipelines using Databricks (PySpark), ADF, Logic Apps, and Airflow.
• Develop end-to-end orchestration frameworks integrating Airflow DAGs with ADF and Logic Apps.
• Implement advanced orchestration patterns (event-driven, micro-batch, hybrid scheduling).
• Ensure pipeline dependency management, execution reliability, and operational excellence.
• Build high-performance ETL/ELT pipelines using Databricks and Delta Lake architecture.
• Implement data observability, monitoring, and alerting across workflows.
• Optimize pipelines for performance, scalability, and cost efficiency.
• Integrate pipelines with ADLS Gen2, Blob Storage, and event triggers.
• Implement CI/CD for pipelines using GitHub Actions or equivalents.
• Ensure data quality, governance, and security compliance.
• Collaborate with data science teams for ML/GenAI data readiness.
• Mentor junior engineers and drive best practices in orchestration.

Skills

Databricks

PySpark

Python

SQL

Orchestration

CI/CD

Monitoring

Data pipelines

Education

7+ years data engineering

Tools

Delta Lake

Azure Data Factory

Azure Logic Apps

Airflow

Job description

Role Overview

We are seeking a Senior Data Engineer to drive cloud data modernization and build scalable, AI-ready data platforms. This role emphasizes expertise in Databricks, PySpark, Azure Data Factory, Logic Apps, and Airflow, with a strong focus on orchestration, pipeline reliability, and end-to-end data workflow management.

Role Overview

We are seeking a Senior Data Engineer to drive cloud data modernization and build scalable, AI-ready data platforms. This role emphasizes expertise in Databricks, PySpark, Azure Data Factory, Logic Apps, and Airflow, with a strong focus on orchestration, pipeline reliability, and end-to-end data workflow management.

Key Responsibilities

• Design, build, and maintain scalable data pipelines using Databricks (PySpark), ADF, Logic Apps, and Airflow
• Develop and manage end-to-end orchestration frameworks integrating Airflow DAGs with ADF and Logic Apps
• Implement advanced workflow orchestration patterns (event-driven, micro-batch, hybrid scheduling)
• Ensure pipeline dependency management, execution reliability, and operational excellence
• Build high-performance ETL/ELT pipelines using Databricks and Delta Lake architecture
• Implement data observability, monitoring, and alerting mechanisms across workflows
• Optimize pipelines and workflows for performance, scalability, and cost efficiency
• Integrate pipelines with Azure services such as ADLS Gen2, Blob Storage, and event triggers
• Implement CI/CD for pipelines and workflows using GitHub Actions or equivalent
• Ensure data quality, governance, and security compliance
• Collaborate with data science teams for ML/GenAI data readiness
• Mentor junior engineers and drive best practices for orchestration and pipeline design.

Required Qualifications

• 7+ years of experience in data engineering
• Highly proficient in Databricks (PySpark, Delta Lake), Azure Data Factory (ADF), Azure Logic Apps, and Apache Airflow
• Exposure to AI/ML and Llm data pipelines and MLflow
• Strong programming skills in Python and advanced SQL
• Experience in building orchestrated data platforms with multi-tool integration
• Strong understanding of ETL/ELT patterns and orchestration frameworks
• Experience in cloud-native data architectures (Azure preferred)
• Hands-on experience with monitoring, logging, and pipeline reliability

(ref:hirist.tech)