Senior Data Engineer

Datamatics; Bangalore Rural; On-site; INR 1,500,000 - 2,100,000; Full time

Job summary

Datamatics in Bangalore is seeking an experienced Data Engineer to design and maintain large-scale ETL pipelines using Python, PySpark, and SQL. You will work with Azure Databricks, Data Factory, SQL Server, and Key Vault to implement robust data solutions.

The role emphasizes CI/CD with GitHub Actions, containerization with OpenShift and Helm, and building dashboards in Grafana to monitor data quality and pipeline health.

Qualifications

• Minimum 6+ years of relevant experience in data engineering and ETL.
• Strong proficiency in Python and PySpark for large-scale data processing.
• Expertise in SQL, data warehousing, and schema design.
• Hands-on Azure analytics stack: Databricks, Data Factory, SQL Server, and Key Vault.
• Experience with CI/CD pipelines (GitHub Actions) for testing, building, and deployment.
• Containerization/orchestration with OpenShift and Helm.
• Experience configuring Grafana dashboards for data quality and pipeline health.

Skills

Python

PySpark

SQL

CI/CD

Grafana

GitHub Actions

Kubernetes

OpenShift

Helm

Azure Databricks

Tools

Azure Databricks

Azure Data Factory

Azure SQL Server

Azure Key Vault

GitHub Actions

OpenShift

Helm

Job description

6+ Years

Required Core Skills & Qualifications:

• Programming & Data Processing: Strong proficiency in Python and PySpark for large-scale data processing and ETL development.
• Data Warehousing & SQL: Expertise in SQL for complex querying, data manipulation, and schema design.(Optional, but highly preferred): Proven experience in SQL optimization and performance tuning.
• ETL Development: Demonstrable experience in designing, building, and maintaining robust ETL/ELT data pipelines.
• Cloud Data Platform (Azure Focus):Hands-on experience with Azure Databrick. Proficiency with core Azure Analytics Services including Azure Data Factory, Azure SQL Server, and Azure Key Vault.
• DevOps & CI/CD: Experience implementing CI/CD pipelines from GitHub (including GitHub Actions) for automated testing (unit tests), build, and deployment processes.
• Containerization & Orchestration: Familiarity and practical experience with OpenShift (setup, deployment-ready configurations, and management).Experience with HELM for deploying applications on Kubernetes/OpenShift.
• Monitoring & Observability: Experience in setting up and configuring Grafana for dashboards to monitor data quality and pipeline health.