AI Data Engineer

SG Analytics; Mumbai; Hybrid; INR 1,500,000 - 2,700,000; Full time

Job summary

SG Analytics in Mumbai seeks an experienced Data Engineering Architect to design and scale enterprise lakehouse solutions on Azure Databricks, ADLS Gen2, Delta Lake, and medallion architecture. You will build production-grade ETL pipelines with PySpark, Spark SQL, and Azure Data Factory, and drive governance and cost-efficient compute strategies.

The role requires 4+ years of hands-on experience building data pipelines and a proven track record with Azure lakehouse architectures.

Qualifications

• 4+ years of hands-on experience building production data pipelines and data platforms.
• Proven track record designing lakehouse architectures on Azure cloud infrastructure.

Responsibilities

• Architect and scale enterprise lakehouse solutions using Azure Databricks, ADLS Gen2, Delta Lake, and medallion architecture (Bronze, Silver, Gold).
• Design, build, and maintain production-grade batch and streaming ELT/ETL pipelines using PySpark, Spark SQL, Azure Data Factory, and Databricks Workflows.
• Develop configuration-driven, metadata-based ingestion frameworks to rapidly onboard structured, semi-structured, and unstructured data sources.
• Curate and optimize gold-layer data products, vector search indexes, and embeddings via Azure AI Search to support Retrieval-Augmented Generation (RAG) and AI agents.
• Enforce end-to-end data governance, access management, and data lineage utilizing Databricks Unity Catalog and Microsoft Purview to ensure strict compliance (Section 7216, PCAOB, NIST AI RMF).
• Implement performance tuning, Liquid Clustering, Z-Ordering, file compaction, and right-sized compute strategies to minimize platform compute costs.
• Establish CI/CD pipelines, automated data quality testing, data contracts, and proactive observability (freshness, volume, schema drift).
• Collaborate with AI developers, architects, and cross-functional teams to integrate point solutions into shared, scalable platform services.

Skills

Azure Databricks

ADLS Gen2

Azure Data Factory

Synapse

Python

PySpark

Spark SQL

SQL

Delta Lake

Parquet

GitHub Actions

Terraform

Bicep

Databricks Unity Catalog

Microsoft Purview

CI/CD

Terraform/Bicep

Tools

Azure DevOps

GitHub Actions

Terraform

Bicep

Job description

Roles and Responsibilities

• Architect and scale enterprise lakehouse solutions using Azure Databricks, ADLS Gen2, Delta Lake, and medallion architecture (Bronze, Silver, Gold).
• Design, build, and maintain production-grade batch and streaming ELT/ETL pipelines using PySpark, Spark SQL, Azure Data Factory, and Databricks Workflows.
• Develop configuration-driven, metadata-based ingestion frameworks to rapidly onboard structured, semi-structured, and unstructured data sources.
• Curate and optimize gold-layer data products, vector search indexes, and embeddings via Azure AI Search to support Retrieval-Augmented Generation (RAG) and AI agents.
• Enforce end-to-end data governance, access management, and data lineage utilizing Databricks Unity Catalog and Microsoft Purview to ensure strict compliance (Section 7216, PCAOB, NIST AI RMF).
• Implement performance tuning, Liquid Clustering, Z-Ordering, file compaction, and right-sized compute strategies to minimize platform compute costs.
• Establish CI/CD pipelines, automated data quality testing, data contracts, and proactive observability (freshness, volume, schema drift).
• Collaborate with AI developers, architects, and cross-functional teams to integrate point solutions into shared, scalable platform services.

Preferred Candidate Profile

Experience

• 4+ years of hands‑on experience building production data pipelines and data platforms.
• Proven track record designing lakehouse architectures on Azure cloud infrastructure.

Technical Expertise

• Core Cloud/Data: Azure Databricks, ADLS Gen2, Azure Data Factory, Synapse.
• Languages & Querying: Strong proficiency in Python, PySpark, Spark SQL, and SQL.
• Storage Formats: Delta Lake, Parquet, Avro, JSON.
• AI Grounding: Vector Search, RAG Pipelines, Azure AI Search, Embeddings, Chunking.
• Governance & DevOps: Databricks Unity Catalog, Microsoft Purview, CI/CD (Azure DevOps/GitHub Actions), Infrastructure-as-Code (Terraform/Bicep).

Mandatory : Candidate must be willing to relocate near to Goregaon, Mumbai Location.