Data Engineer

Tekskills; Surat; Hybrid; INR 4,000,000 - 7,000,000; Full time

Job summary

Tekskills is seeking a seasoned data engineer to design, develop, and optimize scalable data pipelines and ETL processes using Azure Data Services and Databricks. You will collaborate with cross-functional teams in an Agile environment to deliver high-quality data solutions.

The ideal candidate has 10+ years of experience, deep Python expertise, and hands-on experience with Databricks, PySpark, Git, and Azure cloud platforms.

Qualifications

• 10+ years of relevant experience.
• Strong Python programming skills.
• Experience with Databricks, PySpark and Git.
• Experience with Azure cloud platform.

Responsibilities

• Design, develop, and optimize scalable data pipelines and ETL processes using Azure Data Services and Databricks.
• Implement data transformations and analytics using PySpark and Python.
• Configure and manage Databricks Unity Catalog for secure data governance and access control.
• Orchestrate workflows using Apache Airflow for scheduling and automation.
• Collaborate with cross-functional teams to deliver high-quality data solutions in an Agile environment.
• Set up and maintain CI/CD pipelines for data workflows and deployments.
• Manage version control and branching strategies using Git.
• Write and execute unit tests to ensure code quality and reliability.
• Monitor, troubleshoot, and optimize data processes for performance and cost efficiency

Skills

Python expertise

Tools

Databricks

PySpark

Git

Azure Cloud

Job description

JOB Description:

Experience: 10+ Years
Location: Pan India.

Mandatory skills

Deep expertise in python
Experience with Databricks, pyspark and Git.
Experience with Azure cloud platform.

detailed JD

• \"Design, develop, and optimize scalable data pipelines and ETL processes using Azure Data Services and Databricks.


• Implement data transformations and analytics using PySpark and Python.


• Configure and manage Databricks Unity Catalog for secure data governance and access control.


• Orchestrate workflows using Apache Airflow for scheduling and automation.


• Collaborate with cross-functional teams to deliver high-quality data solutions in an Agile environment.


• Set up and maintain CI/CD pipelines for data workflows and deployments.


• Manage version control and branching strategies using Git.


• Write and execute unit tests to ensure code quality and reliability.


• Monitor, troubleshoot, and optimize data processes for performance and cost efficiency