Data Engineer

Tekskills; Ahmedabad District; Hybrid; INR 1,200,000 - 2,100,000; Full time

Job summary

Tekskills in Pan India is seeking a senior data engineer to design, develop, and optimize scalable data pipelines using Azure Data Services and Databricks. You will implement data transformations with PySpark and Python, manage Databricks Unity Catalog for data governance, and orchestrate workflows with Airflow.

Collaborating in an Agile environment, you will also set up CI/CD pipelines for data workloads and maintain Git version control to ensure reliability and cost efficiency.

Qualifications

• Deep expertise in Python.
• Experience with Databricks, PySpark and Git.
• Experience with Azure cloud platform.

Responsibilities

• Design, develop, and optimize scalable data pipelines and ETL processes using Azure Data Services and Databricks.
• Implement data transformations and analytics using PySpark and Python.
• Configure and manage Databricks Unity Catalog for secure data governance and access control.
• Orchestrate workflows using Apache Airflow for scheduling and automation.
• Collaborate with cross-functional teams to deliver high-quality data solutions in an Agile environment.
• Set up and maintain CI/CD pipelines for data workflows and deployments.
• Manage version control and branching strategies using Git.
• Write and execute unit tests to ensure code quality and reliability.
• Monitor, troubleshoot, and optimize data processes for performance and cost efficiency

Skills

Python

Databricks

PySpark

Git

Azure

Job description

JOB Description:

Experience: 10+ Years

Location: Pan India

Mandatory skills

• Deep expertise in python
• Experience with Databricks, pyspark and Git.
• Experience with Azure cloud platform.

Job Details

• Design, develop, and optimize scalable data pipelines and ETL processes using Azure Data Services and Databricks.
• Implement data transformations and analytics using PySpark and Python.
• Configure and manage Databricks Unity Catalog for secure data governance and access control.
• Orchestrate workflows using Apache Airflow for scheduling and automation.
• Collaborate with cross-functional teams to deliver high-quality data solutions in an Agile environment.
• Set up and maintain CI/CD pipelines for data workflows and deployments.
• Manage version control and branching strategies using Git.
• Write and execute unit tests to ensure code quality and reliability.
• Monitor, troubleshoot, and optimize data processes for performance and cost efficiency