Technical Lead -Data Engineer - Azure

embIND Embitel Technologies (India) Private Limited; Bengaluru; On-site; INR 800,000 - 1,400,000; Full time

Job summary

embIND Embitel Technologies (India) Private Limited seeks a data engineer to design and implement scalable data pipelines using Azure Data Factory and Databricks. The role involves developing ETL workflows using Python, collaborating with DevOps for CI/CD management, and ensuring data quality.

The ideal candidate will have a strong background in Azure services and experience with SQL/NoSQL databases. This position offers opportunities to work on innovative data solutions in a collaborative environment.

Qualifications

• Experience in designing and implementing data pipelines with Azure Data Factory.
• Proficient in Python or similar programming languages for ETL processes.
• Familiarity with SQL and NoSQL databases.

Responsibilities

• Design and build secure data pipelines using Azure services.
• Develop and maintain ETL workflows and data transformation.
• Collaborate with teams to manage CI/CD pipelines.

Skills

Azure Data Factory

Azure Databricks

Python

SQL

NoSQL databases

DevOps

ETL

Tools

Azure Functions

Azure DevOps

API Gateway

Job description

Responsibilities

• Design and implement scalable and secure data pipelines using Azure Data Factory and Azure Databricks (Spark).
• Utilize Azure cloud services such as Blob Storage, Azure Functions, and API Gateway to build and maintain robust data solutions.
• Develop and maintain code using Python or similar programming languages to support ETL workflows and data transformation processes.
• Collaborate with DevOps teams to set up and manage CI/CD pipelines using Azure DevOps or equivalent toolsets.
• Ensure high code and data quality standards through performance tuning, testing, and documentation.
• Work across the full development lifecycle, from design through deployment and maintenance of distributed systems.
• Leverage knowledge of SQL and NoSQL databases to design effective data models and ensure data reliability.
• Apply Object-Oriented Programming and ETL best practices to develop modular and reusable components.
• Troubleshoot and resolve cloud networking and connectivity issues in collaboration with infrastructure teams.
• Translate complex technical problems into simple and scalable solutions, aligning with architectural best practices.
• Advocate for and implement data governance, monitoring, and quality frameworks to ensure the integrity of business-critical data.