Data Engineer

Tekskills; Chandigarh; Hybrid; INR 3,000,000 - 6,000,000; Full time

Job summary

Tekskills is seeking a Senior Data Engineer with 10+ years of experience to design and optimize scalable data pipelines using Azure Data Services and Databricks.

You will implement data transformations with PySpark and Python, configure Unity Catalog for governance, orchestrate Airflow workflows, and set up CI/CD for data deployments in an Agile environment.

This nationwide remote role requires collaboration with cross-functional teams to deliver cost-efficient, high-performance data solutions.

Qualifications

• 10+ years of data engineering experience.
• Experience with Azure cloud platform and Databricks.
• Proficiency in PySpark and Python.

Responsibilities

• Design, develop, and optimize scalable data pipelines and ETL processes using Azure Data Services and Databricks.
• Implement data transformations and analytics using PySpark and Python.
• Configure and manage Databricks Unity Catalog for secure data governance and access control.
• Orchestrate workflows using Apache Airflow for scheduling and automation.
• Collaborate with cross-functional teams to deliver high-quality data solutions in an Agile environment.
• Set up and maintain CI/CD pipelines for data workflows and deployments.
• Manage version control and branching strategies using Git.
• Write and execute unit tests to ensure code quality and reliability.
• Monitor, troubleshoot, and optimize data processes for performance and cost efficiency

Skills

Python

Tools

Azure cloud

Databricks

PySpark

Git

Job description

JOB Description:

Experience: 10+ Years

Location: Pan India

Mandatory skills

• Deep expertise in python
• Experience with Databricks, pyspark and Git.
• Experience with Azure cloud platform.

Detailed JD

• Design, develop, and optimize scalable data pipelines and ETL processes using Azure Data Services and Databricks.
• Implement data transformations and analytics using PySpark and Python.
• Configure and manage Databricks Unity Catalog for secure data governance and access control.
• Orchestrate workflows using Apache Airflow for scheduling and automation.
• Collaborate with cross-functional teams to deliver high-quality data solutions in an Agile environment.
• Set up and maintain CI/CD pipelines for data workflows and deployments.
• Manage version control and branching strategies using Git.
• Write and execute unit tests to ensure code quality and reliability.
• Monitor, troubleshoot, and optimize data processes for performance and cost efficiency