Opening For Data architecture

Tekskills Inc.; Bengaluru; In loco; INR 1.200.000 - 1.800.000; Tempo pieno

Descrizione del lavoro

Tekskills Inc. in Bengaluru seeks a data engineer to design, develop, and optimize scalable data pipelines and ETL processes using Azure Data Services and Databricks.

You will implement data transformations with PySpark and Python, set up Unity Catalog governance, orchestrate workflows with Airflow, and maintain CI/CD pipelines for data workloads in an Agile environment. Collaborate with cross-functional teams, manage Git versioning, write unit tests, and monitor performance and cost to deliver

Competenze

• Experience designing scalable data pipelines and ETL using Azure Data Services and Databricks.
• Experience implementing data transformations with PySpark and Python.
• Experience with data governance and secure access using Unity Catalog.
• Experience orchestrating workflows with Apache Airflow in Agile environments.
• Experience setting up CI/CD pipelines for data workloads.

Mansioni

• Design, develop, and optimize scalable data pipelines and ETL processes.
• Implement data transformations using PySpark and Python.
• Configure Unity Catalog for secure data governance and access control.
• Orchestrate workflows with Apache Airflow for scheduling and automation.
• Collaborate with cross-functional teams in an Agile environment.
• Set up and maintain CI/CD pipelines for data workflows.
• Manage version control with Git and write unit tests.
• Monitor and optimize data processes for performance and cost.

Conoscenze

Azure Data Services

Databricks

PySpark

Python

Airflow

CI/CD

Git

Unit testing

Data pipelines

ETL

Strumenti

Databricks

Airflow

Unity Catalog

Git

CI/CD pipelines

Descrizione del lavoro

\"

• Design, develop, and optimize scalable data pipelines and ETL processes using Azure Data Services and Databricks.


• Implement data transformations and analytics using PySpark and Python.


• Configure and manage Databricks Unity Catalog for secure data governance and access control.


• Orchestrate workflows using Apache Airflow for scheduling and automation.


• Collaborate with cross-functional teams to deliver high-quality data solutions in an Agile environment.


• Set up and maintain CI/CD pipelines for data workflows and deployments.


• Manage version control and branching strategies using Git.


• Write and execute unit tests to ensure code quality and reliability.


• Monitor, troubleshoot, and optimize data processes for performance and cost efficiency.

\"

Location: Bangalore,Hyd,Pune,Mysore,Chennai