Senior Data Engineer

Toppan Merril; Chennai; On-site; INR 1,000,000 - 1,500,000; Full time

Job summary

A leading data solutions company based in Chennai, India is seeking a skilled Data Engineer to develop and optimize data pipelines using Azure Databricks and Apache Spark. The ideal candidate will have strong expertise in building distributed data pipelines and proficiency in Python. Responsibilities include ensuring data quality and implementing governance policies using Unity Catalog, alongside automating workflows and collaborating with cross-functional teams.

Qualifications

• Strong experience in building distributed data pipelines.
• Proficiency in writing optimized and maintainable Python code.
• Hands-on experience implementing data governance and access controls.

Responsibilities

• Build, test, and maintain ETL/ELT data pipelines using Azure Databricks.
• Design and enforce access control policies using Unity Catalog.
• Develop CI/CD pipelines for data workflows using Azure Databricks Workflows.

Skills

Azure Databricks & Apache Spark (PySpark)

Python

Unity Catalog

SQL

Delta Lake

Workflow Orchestration

CI/CD & Infrastructure as Code (IaC)

Security & Compliance

Tools

Azure Data Factory

Databricks CLI

MLflow

Job description

# Job Description:**Responsibilities:*** **Develop & Optimize Data Pipelines** + Build, test, and maintain **ETL/ELT** data pipelines using Azure **Databricks & Apache Spark (PySpark)**. + Optimize **performance and cost-efficiency** of Spark jobs. + Ensure data quality through validation, monitoring, and alerting mechanisms. + Understand cluster types, configuration, and use-case for serverless* **Implement Unity Catalog for Data Governance** + Design and enforce **access control policies** using Unity Catalog. + Manage **data lineage, auditing, and metadata governance**. + Enable secure data sharing across teams and external stakeholders.* **Integrate with Cloud Data Platforms** + Work with **Azure Data Lake Storage / Azure Blob Storage/ Azure Event Hub** to integrate Databricks with cloud-based **data lakes, data warehouses, and event streams**. + Implement **Delta Lake** for scalable, ACID-compliant storage.* **Automate & Orchestrate Workflows** + Develop **CI/CD** pipelines for data workflows using **Azure** **Databricks Workflows or Azure Data Factory**. + Monitor and troubleshoot failures in **job execution and cluster performance**.* **Collaborate with Stakeholders** + Work with **Data Analysts, Scientists, and Business Teams** to understand requirements. + Translate business needs into **scalable data engineering solutions**.* API expertise + Ability to pull data from a wide variety of APIs using different strategies and methods**Required Skills & Experience:*** **Azure Databricks & Apache Spark (PySpark)** – Strong experience in building **distributed data pipelines**.* **Python** – Proficiency in writing optimized and maintainable Python code for data engineering.* **Unity Catalog** – Hands-on experience implementing **data governance, access controls, and lineage tracking**.* **SQL** – Strong knowledge of SQL for data transformations and optimizations.* **Delta Lake** – Understanding of **time travel, schema evolution, and performance tuning**.* **Workflow Orchestration** – Experience with **Azure** **Databricks Jobs or Azure Data Factory**.* **CI/CD & Infrastructure as Code (IaC)** – Familiarity with**Databricks CLI, Databricks DABs, and DevOps principles**.* **Security & Compliance** – Knowledge of **IAM, role-based access control (RBAC), and encryption**.**Preferred Qualifications:*** Experience with **MLflow** for model tracking & deployment in Databricks.* Familiarity with **streaming technologies** (Kafka, Delta Live Tables, Azure Event Hub, Azure Event Grid).* Hands-on experience with **dbt (Data Build Tool)** for modular ETL development.* Certification in **Databricks, Azure** is a plus.* Experience with Azure Databricks Lakehouse connectors for SalesForce and SQL Server* Experience with Azure Synapse Link for Dynamics, dataverse* Familiarity with other data pipeline strategies, like Azure Functions, Fabric, ADF, etc**Soft Skills:*** Strong **problem-solving** and debugging skills.* Ability to **work independently and in teams**.* Excellent **communication and documentation** skills.