Data Engineer

Tekskills; Nagpur District; Hybrid; INR 2,500,000 - 4,500,000; Full time

Job summary

Tekskills is seeking a senior data engineer to design and optimize scalable data pipelines within Azure Data Services and Databricks. You will implement data transformations with PySpark and Python, govern data using Unity Catalog, and automate workflows with Apache Airflow.

The role requires a strong background in data engineering, 10+ years of experience, and hands-on expertise with Azure cloud, Databricks, and Git. Join a cross-functional Agile team driving cost-efficient data solutions.

Qualifications

• 10+ years of experience in data engineering or related field.
• Deep expertise in Python programming.
• Experience with Databricks, PySpark and Git.
• Experience with Azure cloud platform.

Responsibilities

• Design, develop, and optimize scalable data pipelines and ETL processes using Azure Data Services and Databricks.
• Implement data transformations and analytics using PySpark and Python.
• Configure and manage Databricks Unity Catalog for secure data governance and access control.
• Orchestrate workflows using Apache Airflow for scheduling and automation.
• Collaborate with cross-functional teams to deliver high-quality data solutions in an Agile environment.
• Set up and maintain CI/CD pipelines for data workflows and deployments.
• Manage version control and branching strategies using Git.
• Write and execute unit tests to ensure code quality and reliability.
• Monitor, troubleshoot, and optimize data processes for performance and cost efficiency.

Skills

Python

Tools

Azure

Databricks

PySpark

Git

Airflow

Job description

JOB Description:

Experience: 10+ Years
Location: Pan India.

Mandatory skills

Deep expertise in python
Experience with Databricks, pyspark and Git.
Experience with Azure cloud platform.

detailed JD

• Design, develop, and optimize scalable data pipelines and ETL processes using Azure Data Services and Databricks.
• Implement data transformations and analytics using PySpark and Python.
• Configure and manage Databricks Unity Catalog for secure data governance and access control.
• Orchestrate workflows using Apache Airflow for scheduling and automation.
• Collaborate with cross-functional teams to deliver high-quality data solutions in an Agile environment.
• Set up and maintain CI/CD pipelines for data workflows and deployments.
• Manage version control and branching strategies using Git.
• Write and execute unit tests to ensure code quality and reliability.
• Monitor, troubleshoot, and optimize data processes for performance and cost efficiency