Azure Engineer

Tech Mahindra; Bengaluru; On-site; INR 1,800,000 - 2,600,000; Full time

Job summary

Tech Mahindra Bengaluru is seeking a data engineer to design, build, and optimize scalable ETL/ELT pipelines using Python and PySpark, primarily on Azure Databricks and Azure Data Factory. You will handle data ingestion from diverse sources, transform data into clean formats, and ensure high pipeline reliability in a cloud-first environment.

You will implement data quality monitoring with Grafana, develop SQL data models, and collaborate with data scientists and other engineers.

Qualifications

• Strong proficiency in Python and PySpark for large-scale data processing.
• Expertise in SQL for complex querying and data modeling.
• Experience designing and maintaining robust ETL/ELT pipelines.
• Hands-on Azure analytics: Azure Databricks, Data Factory, SQL Server, Key Vault.
• DevOps experience: CI/CD with GitHub Actions.
• OpenShift/Kubernetes with HELM deployment experience.
• Monitoring and observability using Grafana.

Responsibilities

• Design, build, and optimize scalable ETL/ELT data pipelines.
• Develop and manage data ingestion from various sources.
• Implement data quality tests and dashboards for monitoring.
• Contribute to OpenShift infrastructure setup and deployment readiness.
• Advance CI/CD automation with GitHub Actions for data solutions.
• Model and optimize data in SQL Server and other stores.
• Collaborate with data scientists and engineers and document workflows.

Skills

Python and PySpark

SQL for data modeling

ETL/ELT pipeline design

Azure data platform

CI/CD with GitHub Actions

OpenShift / Helm

Monitoring with Grafana

Tools

Azure Databricks

Azure Data Factory

Azure SQL Server

OpenShift

Helm

Grafana

GitHub Actions

Job description

• Data Pipeline Development: Design, build, and optimize robust, scalable, and efficient ETL/ELT data pipelines using Python and PySpark, primarily within Azure Databricks and Azure Data Factory.
• Data Ingestion & Processing: Develop and manage processes for ingesting data from various sources (e.g., transactional databases, APIs, streaming sources) and transforming it into clean, usable formats for downstream consumption.
• Data Quality & Monitoring: Implement comprehensive unit and integration test coverage for data pipelines. Establish and maintain monitoring, alerting, and dashboarding solutions (e.g., Grafana) for data quality, pipeline health, and performance.
• Cloud Infrastructure Management (OpenShift/Azure): Contribute to the setup, configuration, and maintenance of data-related infrastructure on OpenShift, ensuring deployment readiness and leveraging tools like HELM for application packaging and deployment.
• CI/CD & Automation: Drive CI/CD best practices using GitHub Actions, ensuring automated testing (unit tests), build, and deployment processes for data solutions to environments like OpenShift.
• SQL & Data Modeling: Develop and optimize complex SQL queries for data extraction, transformation, and loading. Apply strong data modeling principles for efficient data storage and retrieval in SQL Server and other data stores.
• Azure Ecosystem Leverage: Utilize a broad range of Azure data and analytics services, including Azure Data Factory, Azure Databricks, Azure SQL Server, Azure Key Vault, and others to build comprehensive data solutions.
• Performance Optimization: Proactively identify and resolve performance bottlenecks in data pipelines and databases through query optimization, indexing strategies, and efficient data processing techniques.
• Collaboration & Documentation: Work closely with data scientists, analysts, and other engineering teams to understand data requirements. Create clear and concise documentation for data pipelines, architecture, and processes.

Required Core Skills & Qualifications:

• Programming & Data Processing: Strong proficiency in Python and PySpark for large-scale data processing and ETL development.
• Data Warehousing & SQL: Expertise in SQL for complex querying, data manipulation, and schema design.(Optional, but highly preferred): Proven experience in SQL optimization and performance tuning.
• ETL Development: Demonstrable experience in designing, building, and maintaining robust ETL/ELT data pipelines.
• Cloud Data Platform (Azure Focus):Hands-on experience with Azure Databrick. Proficiency with core Azure Analytics Services including Azure Data Factory, Azure SQL Server, and Azure Key Vault.
• DevOps & CI/CD: Experience implementing CI/CD pipelines from GitHub (including GitHub Actions) for automated testing (unit tests), build, and deployment processes.
• Containerization & Orchestration: Familiarity and practical experience with OpenShift (setup, deployment-ready configurations, and management).Experience with HELM for deploying applications on Kubernetes/OpenShift.
• Monitoring & Observability: Experience in setting up and configuring Grafana for dashboards to monitor data quality and pipeline health.