Azure Data Engineer

Lancesoft; Bengaluru; On-site; INR 1,200,000 - 1,800,000; Full time

Job summary

Lancesoft is seeking an experienced Azure Data Engineer in Bengaluru to design, develop, and maintain scalable data pipelines using Microsoft Azure services. The role requires expertise in data integration, ETL/ELT, SQL, and cloud data platforms to support analytics and data warehousing initiatives.

Responsibilities include building data pipelines with ADF, leveraging Databricks for processing, and tuning performance while ensuring data quality, security, and governance.

Qualifications

• Experience designing, developing, and maintaining scalable data pipelines on Azure.
• Strong SQL skills and Python/PySpark for data processing.
• Hands-on with data integration, ETL/ELT concepts, and data warehousing.
• Familiarity with data governance, security, and IAM on Azure.
• Experience with REST APIs and data formats like JSON, CSV, Parquet, Avro.

Responsibilities

• Design and develop scalable data pipelines and ETL/ELT workflows on Azure.
• Develop data integration solutions using Azure Data Factory.
• Work with Azure Databricks for data processing and analytics.
• Write complex SQL queries, stored procedures, and views.
• Build data solutions using ADLS Gen2 and Synapse Analytics.
• Implement incremental loads, CDC, partitioning, and performance tuning.
• Integrate data from databases, APIs, files, and cloud apps.
• Monitor, troubleshoot pipelines, and resolve production issues.
• Implement security, access control, and governance on Azure.
• Collaborate with Data Scientists, BI Developers, Architects, stakeholders.
• Set up CI/CD for data workflows using Azure DevOps.
• Optimize pipelines and cloud resource costs.

Job description

Role & responsibilities

Job Summary:

We are looking for an experienced Azure Data Engineer to design, develop, and maintain scalable data pipelines and data solutions using Microsoft Azure services. The candidate should have strong experience in data integration, ETL/ELT, cloud data platforms, SQL, and Azure data engineering technologies.

Key Responsibilities

• Design and develop scalable data pipelines and ETL/ELT workflows on Azure.
• Develop data integration solutions using Azure Data Factory (ADF).
• Work with Azure Databricks for data processing, transformation, and analytics.
• Develop complex SQL queries, stored procedures, views, and data transformations.
• Build and maintain data solutions using Azure Data Lake Storage (ADLS Gen2).
• Work with Azure Synapse Analytics for data warehousing and analytical workloads.
• Perform data cleansing, transformation, validation, and quality checks.
• Implement incremental data loads, CDC, partitioning, and performance optimization.
• Integrate data from various sources such as databases, APIs, files, and cloud applications.
• Monitor and troubleshoot data pipelines and resolve production issues.
• Implement security, access control, and governance using Azure services.
• Collaborate with Data Scientists, BI Developers, Architects, and business stakeholders.
• Implement CI/CD for data engineering workflows using Azure DevOps/Git.
• Optimize data pipelines and cloud resources for performance and cost.

Required Skills

• Strong experience in Azure Data Engineering.
• Hands-on experience with:
  • Azure Data Factory (ADF)
  • Azure Databricks
  • Azure Data Lake Storage Gen2
  • Azure Synapse Analytics
• Strong SQL skills.
• Good knowledge of Python/PySpark.
• Strong understanding of ETL/ELT concepts and data warehousing.
• Experience with data modeling, partitioning, and performance tuning.
• Knowledge of REST APIs and different data formats such as JSON, CSV, Parquet, and Avro.
• Experience with Git and Azure DevOps.
• Understanding of cloud security, IAM, and data governance.

Preferred candidate profile