Azure Data Engineer - Factspan Analytics

Factspan Analytics; Bengaluru; Hybrid; INR 2,000,000 - 2,800,000; Full time

Job summary

Factspan Analytics is seeking a senior Data Engineer to lead Legacy-to-Cloud migrations to Azure and build robust Databricks pipelines using PySpark and Delta Lake.

You will translate Informatica and Mainframe mappings into modern Azure-based transformations, optimize large-scale joins, and implement cost-efficient CI/CD practices with Azure DevOps.

Qualifications

• 6+ years of experience in Data Engineering.
• Bachelor’s or Master’s in Computer Science, Information Systems, or related field.
• Proven track record of full-cycle migrations from on-premise to Azure.
• Expertise in Databricks SQL and Spark core with performance optimization.
• Azure Certification: AZ-204 (Developer) or AZ-305 (Solutions Architect) is a significant advantage.

Responsibilities

• Execute Legacy-to-Cloud migrations end-to-end from on-prem to Azure.
• Develop and optimize PySpark/Delta Live Tables pipelines.
• Map legacy Informatica and Mainframe logic to code-based Azure transformations.
• Tune Databricks clusters for cost and performance; implement CI/CD with Azure DevOps.

Skills

Databricks

PySpark

Delta Lake

SQL

Shell scripting

Big Data

Education

Bachelor's/Master's in CS/IS

Tools

Azure Data Factory

ADLS Gen2

Synapse

Azure DevOps

Databricks

Airflow

Job description

Responsibilities

• Execute Legacy-to-Cloud Migrations: Lead the end-to-end migration of complex datasets from on-premise legacy systems (including Mainframe and Informatica-based workflows) to the Azure.
• Develop Databricks Pipelines: Build and optimize high-throughput ETL/ELT pipelines using PySpark and Delta Live Tables (DLT) to ensure data consistency and reliability.
• Integrate Hybrid Workflows: Map and translate legacy logic (Informatica mappings and Mainframe COBOL/Copybooks) into modern, code-based transformations within Azure Data Factory (ADF) and Databricks.
• Performance Tuning & MLOps: Monitor and tune Databricks clusters for cost-efficiency and performance; implement CI/CD pipelines via Azure DevOps to automate data deployments.

Required Technology stack

• Azure (Data Factory, ADLS Gen2, Synapse, Azure DevOps), Azure Databricks, Apache Spark (PySpark/Scala), Delta Lake, Advanced Python, SQL, and Shell Scripting, ADF Triggers, Databricks Workflows, or Airflow

Good to have

• Informatica PowerCenter (mappings/workflows), Mainframe (DB2, VSAM, JCL).

Experience & Qualifications

• 6+ years of experience in Data Engineering.
• Bachelors or Masters Degree in Computer Science, Information Systems, or a related field.
• Proven track record of a full-cycle migrations from on-premise environments to Azure.
• Expertise in Databricks SQL and Spark core, specifically focusing on performance optimization of large-scale joins and aggregations.
• Azure Certification: AZ-204 (Developer) or AZ-305 (Solutions Architect) is a significant advantage