Data Engineer_Azure

Factspan Analytics; Bengaluru; Hybrid; INR 1,200,000 - 1,800,000; Full time

Job summary

Factspan Analytics is looking for a skilled Data Engineer based in Bengaluru, India. The role involves leading migration projects from legacy systems to Azure, alongside developing and optimizing Databricks ETL pipelines.

The ideal candidate should have 6-10 years of experience in Data Engineering, expertise in Azure services, and a proven record of full migration cycles. Azure certification is a plus.

Qualifications

• 6-10 years of experience in Data Engineering.
• Proven track record of full-cycle migrations to Azure.
• Azure Certification AZ-204 or AZ-305 is advantageous.

Responsibilities

• Lead the end-to-end migration of complex datasets to Azure.
• Build and optimize ETL/ELT pipelines using PySpark.
• Map legacy logic into modern transformations in Azure Data Factory.

Skills

Data Engineering

Azure Data Factory

Databricks

PySpark

SQL

Performance optimization

Education

Bachelor's degree in Computer Science

Master's degree in Computer Science

Tools

Azure DevOps

Apache Spark

Delta Lake

Job description

Responsibilities

• Execute Legacy-to-Cloud Migrations: Lead the end-to-end migration of complex datasets from on-premise legacy systems (including Mainframe and Informatica-based workflows) to the Azure.
• Develop Databricks Pipelines: Build and optimize high-throughput ETL/ELT pipelines using PySpark and Delta Live Tables (DLT) to ensure data consistency and reliability.
• Integrate Hybrid Workflows: Map and translate legacy logic (Informatica mappings and Mainframe COBOL/Copybooks) into modern, code-based transformations within Azure Data Factory (ADF) and Databricks.
• Performance Tuning & MLOps: Monitor and tune Databricks clusters for cost-efficiency and performance; implement CI/CD pipelines via Azure DevOps to automate data deployments.

Required Technology stack

• Azure (Data Factory, ADLS Gen2, Synapse, Azure DevOps), Azure Databricks, Apache Spark (PySpark/Scala), Delta Lake, Advanced Python, SQL, and Shell Scripting, ADF Triggers, Databricks Workflows, or Airflow

Good to have

• Informatica PowerCenter (mappings/workflows), Mainframe (DB2, VSAM, JCL).

Experience & Qualifications

• 6- 10 years of experience in Data Engineering.
• Bachelors or Masters Degree in Computer Science, Information Systems, or a related field.
• Proven track record of a full-cycle migrations from on-premise environments to Azure.
• Expertise in Databricks SQL and Spark core, specifically focusing on performance optimization of large-scale joins and aggregations.
• Azure Certification: AZ-204 (Developer) or AZ-305 (Solutions Architect) is a significant advantage