Opening For Azure data engineer

Tekskills Inc.; Chennai; On-site; INR 2,500,000 - 3,800,000; Full time

Job summary

Tekskills Inc. is seeking an Architect to build and deploy scalable Databricks solutions on Microsoft Azure. You will design data pipelines, ETL processes, and ML workflows while optimizing performance and cost.

You will ensure security and regulatory compliance, manage encryption and access controls, and collaborate with data engineers, scientists, and analysts to optimize data workflows. You will also document architectures and provide training.

Qualifications

• Experience designing and deploying Databricks-based data pipelines on Azure.
• Proficiency in Spark, Delta Lake and data warehousing concepts.
• Strong knowledge of Azure services (Data Factory, Data Lake, SQL Database, Synapse).
• Ability to implement secure, compliant data architectures and encryption.
• Familiarity with DevOps practices and CI/CD pipelines for data workloads.

Responsibilities

• Architect and deploy scalable Databricks solutions on Azure.
• Develop and implement data pipelines, ETL processes, and ML workflows.
• Monitor Databricks performance and optimize configurations for cost-efficiency.
• Ensure data storage, processing, and analysis meet performance targets.
• Enforce security, encryption, access controls, and regulatory compliance.
• Perform maintenance, updates, patches, and backups.
• Provide advanced Databricks support and escalate complex issues.
• Collaborate with data engineers, scientists, and analysts on workflows.
• Integrate Databricks with other Azure services and tools.
• Deliver training on best practices and new features.
• Document architectures, configurations, and procedures for knowledge sharing.

Skills

Databricks

Azure

ETL design

Data pipelines

ML workflows

Security compliance

Agile

Documentation

Team collaboration

Multi-cloud

DevOps practices

Python

SQL

Scala

Delta Lake

Spark

Azure DevOps

Jenkins

GitHub Actions

Azure Data Lake

Azure SQL Database

Cosmos DB

Power BI

Tools

Azure DevOps

Jenkins

GitHub Actions

Azure Data Lake

Azure SQL Database

Delta Lake

Power BI

Azure Synapse Analytics

Cosmos DB

Job description

Architect and deploy scalable Databricks solutions on Azure.

• Develop and implement data pipelines, ETL processes, and machine learning workflows using Databricks and Azure services.
• Monitor Databricks performance and optimize configurations for efficiency and cost-effectiveness.
• Implement best practices for data storage, processing, and analysis to ensure optimal performance.
• Ensure Databricks environments comply with security standards and regulatory requirements.
• Implement and manage data encryption, access controls, and other security measures.
• Perform regular maintenance tasks, including updates, patches, and backups.
• Provide advanced technical support for Databricks-related issues and act as the escalation point for complex problems.
• Collaborate with data engineers, data scientists, and analysts to design and optimize data workflows.
• Work with IT and DevOps teams to integrate Databricks with other Azure services and tools.
• Develop and deliver training sessions for internal teams on Databricks best practices and new features.
• Document architectures, configurations, and procedures for knowledge sharing and future reference.

To be successful in this role, you should meet the following requirements:

• Strong analytical and problem-solving skills.
• Strong technical documentation capabilities.
• Agile framework and delivery lifecycle management.
• Team based development capabilities and ability to manage personal deliverables.
• Experience with multi-cloud environments and hybrid cloud strategies.
• Familiarity with DevOps practices and tools such as Azure DevOps, Jenkins, or GitHub Actions and Azure DevOps
• Databricks Certified Associate Developer or equivalent certification.
• Relevant Azure certifications, such as Azure Data Engineer Associate or Azure Solutions Architect
• In-depth knowledge of Databricks architecture, Spark, and Delta Lake.
• Proficiency in managing and optimizing Databricks clusters and jobs.
• Experience with deploying and managing Databricks on Azure, including integration with Azure Data Lake, Azure SQL Database, and other Azure services.
• Strong programming skills in languages such as Python, SQL, and Scala.
• Familiarity with data warehousing concepts and ETL processes.
• Strong understanding of Azure infrastructure and services, such as Azure Virtual Machines, Azure Storage, and Azure Networking.
• Azure Database and data platform
• Database Services, including Azure SQL and Cosmos DB
• Monitoring, Diagnostics, and Analytics with Azure Monitor
• Azure Data Factory and Data Pipeline Orchestration
• Azure Synapse Analytics and Data Warehousing
• Data Modeling and Database Design
• Power BI and Data Visualization
• MongoDB Experience a plus