Senior Data Engineer (Immediate Joiners Only)

SG Analytics; Pune District, Chennai District; Hybrid; INR 5,000,000 - 7,500,000; Full time

Job summary

SG Analytics in Pune, India, is seeking an experienced engineering lead to design, build, and scale enterprise-grade ELT/ETL pipelines using Python, PySpark, Kafka and Databricks. The role emphasizes hands-on delivery of AI agents and data-driven solutions in banking domains.

You will lead integration of microservices on cloud-native infra with OpenShift or Kubernetes, implement CI/CD, and ensure data governance and regulatory compliance across lifecycles.

Qualifications

• 8+ years in large-scale application development with AI agents in production
• 5+ years of Python and PySpark engineering leadership for high-volume ELT/ETL

Responsibilities

• Design, build, and maintain enterprise-grade ELT and ETL processes using Python, PySpark, Kafka, and Databricks.
• Develop and deploy Gen AI agents using Google ADK, Google Flash 2.5+ LLMs, and HIL architecture.
• Build data federation layers supporting Data Mesh with Starburst for ML/NLP use cases.
• Develop, deploy, and automate microservice integrations on cloud-native infrastructure using OpenShift or Kubernetes with CI/CD pipelines.
• Integrate agentic AI tools (Devin.AI, GitHub Copilot) and Model Context Protocol using advanced prompt engineering.
• Ensure strict data quality, governance, risk assessments, and regulatory compliance across all data lifecycles.

Skills

Python

PySpark

Kafka

Databricks

OpenShift

Kubernetes

CI/CD

Gen AI

Data Mesh

Starburst

NLP

Prompt engineering

Devin.AI

GitHub Copilot

ML/DL

Tools

Databricks

Starburst

OpenShift

GitHub Copilot

Devin.AI

Job description

Roles and Responsibilities

• Design, build, and maintain enterprise-grade ELT and ETL processes handling large data volumes using Python, PySpark, Kafka, and Databricks.
• Develop and deploy Gen AI agents using Google ADK, Google Flash 2.5+ LLMs, and Human-in-the-Loop (HIL) architecture.
• Build data federation layers supporting Data Mesh architectures with Starburst to enable machine learning, deep learning, and NLP use cases.
• Develop, deploy, and automate microservice integrations on cloud-native infrastructure using OpenShift or Kubernetes with automated CI/CD pipelines.
• Integrate agentic AI tools (Devin.AI, GitHub Copilot) and Model Context Protocol (MCP) using advanced prompt engineering to boost development efficiency.
• Ensure strict data quality, governance, risk assessments, and regulatory compliance across all engineered data lifecycles.

Preferred Candidate Profile

• Experience:
  • 8+ years overall experience in large-scale application development with recent hands-on deployment of AI agents into production.
  • 5+ years of experience in a Python and PySpark Engineering Lead role delivering high-volume ELT/ETL processes.
  • Domain background in Banking, Cards, Mortgage, Deposits, or Wealth Management is preferred.