Azure Data Engineer

eInfochips; Indore District, Ahmedabad District; On-site; INR 2,000,000 - 4,000,000; Full time

Job summary

eInfochips in India invites a Senior Data Engineer to design and operate scalable data pipelines on Azure, including batch and streaming ingestion with PySpark/SQL. You will publish curated datasets for analytics and ML, enforce data quality, and collaborate with data teams to ensure governance and security across data platforms.

The role requires 5–8 years of experience with Azure Data Factory, Databricks, Delta Lake, and strong Python/SQL skills.

Qualifications

• Production pipelines with Azure Data Factory and Databricks (PySpark/SQL).
• Medallion Architecture and Delta Lake knowledge (schema evolution, ACID).
• Strong Python and SQL with CI/CD experience (Azure DevOps/GitHub Actions/Jenkins).
• Data modeling/warehousing with Synapse/Snowflake/SQL Server understanding.

Responsibilities

• Design, build, and maintain ETL/ELT pipelines in Azure Data Factory and Databricks across Bronze Silver Gold layers.
• Implement Delta Lake features (ACID, schema evolution, MERGE/upsert, time travel, Z-ORDER).
• Write PySpark and SQL; tune jobs (partitioning, caching, join strategies).
• Create reusable components; manage code in Git; contribute to CI/CD pipelines (Azure DevOps/GitHub Actions/Jenkins).
• Apply data quality checks and monitoring; drift detection and alerting.
• Model data for analytics (star/dimensional); publish to Synapse/Snowflake/SQL Server.
• Uphold governance and security (Purview/Unity Catalog lineage, RBAC, tagging, encryption, PII handling).
• Author documentation/runbooks; support production incidents and root-cause analysis; suggest cost/performance improvements.

Skills

Python

SQL

Azure Databricks

ADF

ETL

PySpark

Education

Bachelor’s/Master’s in CS/IT/Engineering

Tools

Azure Data Factory

Databricks

Delta Lake

Job description

Role: Senior Data Engineer

Years of Experience: 5-8 Years

Location: Ahmedabad, Pune, Indore

Notice Period: Immediate to 30 Days Max

Must Skill: Python, SQL, Azure Databricks, ADF, ETL, Pyspark,

Role Summary

Build and operate scalable, reliable data pipelines on Azure. Develop batch and streaming ingestion, transform data using Databricks (PySpark/SQL), enforce data quality, and publish curated datasets for analytics and ML.

Key Responsibilities

• Design, build, and maintain ETL/ELT pipelines in Azure Data Factory and Databricks across Bronze Silver Gold layers.
• Implement Delta Lake best practices (ACID, schema evolution, MERGE/upsert, time travel, Z-ORDER).
• Write performant PySpark and SQL; tune jobs (partitioning, caching, join strategies).
• Create reusable components; manage code in Git; contribute to CI/CD pipelines (Azure DevOps/GitHub Actions/Jenkins).
• Apply data quality checks (Great Expectations or custom validations), monitoring, drift detection, and alerting.
• Model data for analytics (star/dimensional); publish to Synapse/Snowflake/SQL Server.
• Uphold governance and security (Purview/Unity Catalog lineage, RBAC, tagging, encryption, PII handling).
• Author documentation/runbooks; support production incidents and root-cause analysis; suggest cost/performance improvements.

Must-Have (Mandatory)

• Data Engineering & Pipelines
• Hands-on experience building production pipelines with Azure Data Factory and Databricks (PySpark/SQL).
• Working knowledge of Medallion Architecture and Delta Lake (schema evolution, ACID).
• Programming & Automation
• Strong Python (pandas/PySpark) and SQL.
• Practical Git workflow; experience integrating pipelines into CI/CD (Azure DevOps/GitHub Actions/Jenkins).
• Familiarity with packaging reusable code (e.g., Python wheels) and configuration-driven jobs.
• Data Modeling & Warehousing
• Solid grasp of dimensional modeling/star schemas; experience with Synapse, Snowflake, or SQL Server.
• Data Quality & Monitoring
• Implemented validation checks and alerts; exposure to drift detection and pipeline observability.
• Cloud Platforms (Azure preferred)
• ADLS Gen2, Key Vault, Databricks, ADF basics (linked services, datasets, triggers), environment promotion.
• Data Governance & Security
• Experience with metadata/lineage (Purview/Unity Catalog), RBAC, secrets management, and secure data sharing.
• Understanding of PII/PHI handling and encryption at rest/in transit.
• Collaboration
• Clear communication, documentation discipline, Agile ways of working, and code reviews.

Good-to-Have

• Databricks Asset Bundles (DAB) for environment promotion/infra-as-code style deployments.
• Streaming/real-time: Kafka/Event Hubs; CDC tools (e.g., Debezium, ADF/Synapse CDC).
• MLOps touchpoints: MLflow tracking/registry, feature tables, basic model-inference pipelines.
• Power BI exposure for publishing curated tables and building operational KPIs.
• DataOps practices: automated testing, data contracts, lineage-aware deployments, cost optimization on Azure.
• Certifications: Microsoft Certified Azure Data Engineer Associate (DP-203) or equivalent.

Qualifications

• 46 years of professional experience in data engineering (or equivalent project depth).
• Bachelor’s/Master’s in CS/IT/Engineering or related field (or equivalent practical experience).