Opening For Data architecture

Tekskills Inc.; Chennai; Presencial; INR 1 200 000 - 2 000 000; Tempo integral

Resumo da oferta

Tekskills Inc. in Chennai is seeking a Data Engineer to design, build, and optimize scalable data pipelines using Azure Data Services and Databricks.

You will implement data transformations with PySpark/Python, govern data with Unity Catalog, and automate workflows with Airflow in an Agile team. Responsibilities include CI/CD for data workloads, Git version control, and rigorous unit testing to ensure reliability and cost efficiency.

Qualificações

• Experience designing data pipelines in Azure environments.
• Proficient with Databricks, PySpark and Python.
• Knowledge of data governance and Unity Catalog.
• Familiarity with CI/CD for data workloads.

Responsabilidades

• Design, develop, and optimize scalable data pipelines and ETL processes.
• Implement data transformations and analytics using PySpark and Python.
• Configure Databricks Unity Catalog for secure data governance.
• Orchestrate workflows with Apache Airflow for scheduling.
• Collaborate with cross-functional teams in Agile environment.
• Set up and maintain CI/CD pipelines for data workflows.
• Manage version control and branching with Git.
• Write and execute unit tests for code quality.
• Monitor and optimize data processes for performance and cost.

Conhecimentos

Azure Data Services

Databricks

PySpark

Python

Airflow

CI/CD

Git

Agile

Ferramentas

Databricks Unity Catalog

Airflow

Descrição da oferta de emprego

• Design, develop, and optimize scalable data pipelines and ETL processes using Azure Data Services and Databricks.
• Implement data transformations and analytics using PySpark and Python.
• Configure and manage Databricks Unity Catalog for secure data governance and access control.
• Orchestrate workflows using Apache Airflow for scheduling and automation.
• Collaborate with cross-functional teams to deliver high-quality data solutions in an Agile environment.
• Set up and maintain CI/CD pipelines for data workflows and deployments.
• Manage version control and branching strategies using Git.
• Write and execute unit tests to ensure code quality and reliability.
• Monitor, troubleshoot, and optimize data processes for performance and cost efficiency.

Location: Bangalore,Hyd,Pune,Mysore,Chennai