Data Engineer

Tekskills; Indore District; Hybrid; INR 4,000,000 - 7,000,000; Full time

Job summary

Tekskills in India is seeking a senior data engineer to design, develop, and optimize scalable data pipelines using Azure Data Services, Databricks, and PySpark. You will collaborate with cross-functional teams in an Agile environment to deliver reliable data solutions.

The role emphasizes code quality, CI/CD for data workflows, and strong governance with Git and Unity Catalog, while monitoring performance and cost. Remote, nationwide deployment across Pan India with flexible work arrangements.

Qualifications

• 10+ years of experience in data engineering or similar roles.
• Deep expertise in Python and distributed data processing.
• Strong hands-on experience with Databricks, PySpark, and Git.

Responsibilities

• Design, develop, and optimize scalable data pipelines and ETL processes using Azure Data Services and Databricks.
• Implement data transformations and analytics using PySpark and Python.
• Configure and manage Unity Catalog for secure data governance and access control.
• Orchestrate workflows using Apache Airflow for scheduling and automation.
• Collaborate with cross-functional teams in an Agile environment to deliver high-quality data solutions.
• Set up and maintain CI/CD pipelines for data workflows and deployments.
• Manage version control and branching strategies using Git.
• Write and execute unit tests to ensure code quality and reliability.
• Monitor, troubleshoot, and optimize data processes for performance and cost efficiency

Skills

Python

Databricks

PySpark

Git

Azure Cloud

Tools

Apache Airflow

Azure Data Factory

Job description

JOB Description:

Experience: 10+ Years
Location: Pan India.

Mandatory skills

Deep expertise in python
Experience with Databricks, pyspark and Git.
Experience with Azure cloud platform.

detailed JD

• \"Design, develop, and optimize scalable data pipelines and ETL processes using Azure Data Services and Databricks.


• Implement data transformations and analytics using PySpark and Python.


• Configure and manage Databricks Unity Catalog for secure data governance and access control.


• Orchestrate workflows using Apache Airflow for scheduling and automation.


• Collaborate with cross-functional teams to deliver high-quality data solutions in an Agile environment.


• Set up and maintain CI/CD pipelines for data workflows and deployments.


• Manage version control and branching strategies using Git.


• Write and execute unit tests to ensure code quality and reliability.


• Monitor, troubleshoot, and optimize data processes for performance and cost efficiency