Walk-in | Azure Data Engineer

Lancesoft; Bengaluru; On-site; INR 1,500,000 - 3,500,000; Full time

Job summary

Lancesoft is seeking an experienced Azure Data Engineer in Bengaluru to design, develop, and maintain scalable data pipelines and data solutions using Microsoft Azure services. The role requires strong data integration, ETL/ELT, SQL, and Azure data engineering tech.

You will work with ADF, Databricks, and ADLS Gen2, optimize pipelines, implement security, and collaborate with data scientists and BI teams to deliver reliable data products.

Qualifications

• Experience designing scalable data pipelines on Microsoft Azure.
• Strong SQL, data integration, and data quality skills.
• Proficiency with Python or PySpark for data processing.
• Experience with data warehousing concepts and modeling.
• Familiarity with CI/CD for data workflows.
• Knowledge of REST APIs and common data formats (JSON, CSV, Parquet, Avro).

Responsibilities

• Design and develop scalable data pipelines and ETL/ELT workflows on Azure.
• Develop data integration solutions using Azure Data Factory (ADF).
• Work with Azure Databricks for processing, transformation, and analytics.
• Write complex SQL queries, stored procedures, views, and data transformations.
• Build and maintain data solutions using ADLS Gen2.
• Leverage Azure Synapse Analytics for warehousing and analytics.
• Perform data cleansing, transformation, validation, and quality checks.
• Implement incremental data loads, CDC, partitioning, and performance tuning.
• Integrate data from databases, APIs, files, and cloud apps.
• Monitor, troubleshoot, and resolve production data pipeline issues.
• Apply security, access control, and governance using Azure services.
• Collaborate with Data Scientists, BI Developers, and stakeholders.
• Implement CI/CD for data workflows using Azure DevOps/Git.

Skills

SQL

Python/PySpark

ETL/ELT concepts

Data warehousing

Data modeling

CI/CD concepts

REST APIs

Tools

Azure Data Factory (ADF)

Azure Databricks

Azure Data Lake Storage Gen2

Azure Synapse Analytics

Git

Azure DevOps

Job description

Role & responsibilities

Job Summary:

We are looking for an experienced Azure Data Engineer to design, develop, and maintain scalable data pipelines and data solutions using Microsoft Azure services. The candidate should have strong experience in data integration, ETL/ELT, cloud data platforms, SQL, and Azure data engineering technologies.

Key Responsibilities

• Design and develop scalable data pipelines and ETL/ELT workflows on Azure.
• Develop data integration solutions using Azure Data Factory (ADF).
• Work with Azure Databricks for data processing, transformation, and analytics.
• Develop complex SQL queries, stored procedures, views, and data transformations.
• Build and maintain data solutions using Azure Data Lake Storage (ADLS Gen2).
• Work with Azure Synapse Analytics for data warehousing and analytical workloads.
• Perform data cleansing, transformation, validation, and quality checks.
• Implement incremental data loads, CDC, partitioning, and performance optimization.
• Integrate data from various sources such as databases, APIs, files, and cloud applications.
• Monitor and troubleshoot data pipelines and resolve production issues.
• Implement security, access control, and governance using Azure services.
• Collaborate with Data Scientists, BI Developers, Architects, and business stakeholders.
• Implement CI/CD for data engineering workflows using Azure DevOps/Git.
• Optimize data pipelines and cloud resources for performance and cost.

Required Skills

• Strong experience in Azure Data Engineering.
• Hands-on experience with:
  • Azure Data Factory (ADF)
  • Azure Databricks
  • Azure Data Lake Storage Gen2
  • Azure Synapse Analytics
• Strong SQL skills.
• Good knowledge of Python/PySpark.
• Strong understanding of ETL/ELT concepts and data warehousing.
• Experience with data modeling, partitioning, and performance tuning.
• Knowledge of REST APIs and different data formats such as JSON, CSV, Parquet, and Avro.
• Experience with Git and Azure DevOps.
• Understanding of cloud security, IAM, and data governance.

Preferred candidate profile