AI Data Engineer (Azure)

Careernet; Kolkata District, Pune District, Mumbai; Hybrid; INR 1,200,000 - 2,200,000; Full time

Job summary

Careernet is seeking a seasoned data engineer in Kolkata to design and optimize scalable ETL/ELT pipelines on Azure. You will build production-grade data platforms using Azure Data Factory, Databricks, Python, and Spark, implementing Delta Lake lakehouse models and robust governance across ADLS Gen2.

The role involves handling large-scale data formats (Parquet, Avro, JSON), enabling AI data prep for embeddings and vector search, and integrating data access controls and lineage.

Qualifications

• 4+ years of relevant experience in data engineering, Azure data platforms, and production pipeline delivery.
• Strong hands-on experience with Azure Data Factory, Azure Databricks, Python, SQL, and Spark SQL/PySpark.
• Experience building production data platforms on ADLS Gen2 and working with Delta Lake and lakehouse patterns.
• Ability to handle large-scale data processing with columnar and row-based formats such as Parquet, Avro, and JSON.
• Experience with data governance, lineage, and access control tools such as Unity Catalog and Microsoft Purview.

Responsibilities

• Design and build scalable ETL/ELT pipelines for batch and analytical workloads on Azure.
• Develop and optimize data processing workflows in Azure Databricks using Python, SQL, and Spark.
• Implement lakehouse data models using Delta Lake and medallion architecture patterns.
• Integrate governed data sources and manage access controls, lineage, and cataloging across the platform.
• Support AI data preparation workflows for embeddings, vector search, and RAG-oriented datasets.

Skills

LLM

Azure Databricks

SQL

AI Engineering

Python

Azure Data lake

ETL

Azure ADF

GitHub

DBT

Kafka

Spark

Education

Bachelor's degree

Tools

Spark

Kafka

GitHub

dbt

Structured Streaming

Azure Event Hubs

Debezium

Great Expectations

Delta Live Tables

Terraform

Bicep

Apache Iceberg

Delta UniForm

Azure DevOps

GitHub Actions

Key Vault

Managed Identity

Job description

Key Skills:

LLM, Azure Databricks, SQL, AI Engineering, Python, Azure Data lake, ETL, Azure ADF, GitHub, DBT, Kafka, Spark

Roles and Responsibilities:

• Design and build scalable ETL/ELT pipelines for batch and analytical workloads on Azure.
• Develop and optimize data processing workflows in Azure Databricks using Python, SQL, and Spark.
• Implement lakehouse data models using Delta Lake and medallion architecture patterns.
• Integrate governed data sources and manage access controls, lineage, and cataloging across the platform.
• Support AI data preparation workflows for embeddings, vector search, and RAG-oriented datasets.

Skills Required:

• 4+ years of relevant experience in data engineering, Azure data platforms, and production pipeline delivery.
• Strong hands-on experience with Azure Data Factory, Azure Databricks, Python, SQL, and Spark SQL/PySpark.
• Experience building production data platforms on ADLS Gen2 and working with Delta Lake and lakehouse patterns.
• Ability to handle large-scale data processing with columnar and row-based formats such as Parquet, Avro, and JSON.
• Experience with data governance, lineage, and access control tools such as Unity Catalog and Microsoft Purview.

Good to Have:

• Experience with Spark, Kafka, GitHub, dbt, Structured Streaming, Azure Event Hubs, Debezium, Great Expectations, Delta Live Tables, Terraform, Bicep, Apache Iceberg, Delta UniForm, Azure DevOps, GitHub Actions, Key Vault, and Managed Identity.

Education:

Bachelor's degree in Computer Science, Information Technology, Engineering, or a related discipline.