Data Engineer

SG Analytics; Chennai District; Hybrid; INR 3,000,000 - 5,400,000; Full time

Job summary

SG Analytics seeks a seasoned Senior Data/Software Engineer to design and scale AI agents, ETL pipelines, and data federation for enterprise-grade applications in Chennai. The role emphasizes GenAI integration, data security, and compliant data lifecycles, with leadership in Python, PySpark, and Databricks architectures.

You will drive architecture, implement CI/CD, and collaborate across teams to deliver high-velocity automation and reliable data platforms.

Qualifications

• 8+ years in large-scale software engineering or data platform development.
• 5+ years of hands-on technical lead experience in Python, PySpark, and Databricks pipeline architectures.
• Background in Banking, Financial Services, Retail Products (Cards, Mortgages, Deposits), or Wealth Management risk domains is strongly preferred.

Responsibilities

• Design, develop, and maintain scalable, enterprise-grade AI agents and high-volume ELT/ETL pipelines using Python, PySpark, Databricks, Kafka, and FastAPI.
• Implement and deploy GenAI agents using Google ADK and Google Flash 2.5+ LLMs to power application automation and workflow optimization via Human-in-the-Loop (HIL) design.
• Build and maintain data federation layers for Lambda and Data Mesh architectures using Starburst to enable machine learning, deep learning, and NLP use cases.
• Develop, deploy, and automate resilient microservice integrations supporting data-intensive applications on Kubernetes and OpenShift cloud-native platforms.
• Integrate agentic AI tools such as Devin.AI and GitHub Copilot via Model Context Protocol (MCP) and advanced prompt engineering to maximize development velocity.
• Enforce data quality, security, and risk compliance across the data lifecycle, ensuring adherence to regulatory standards and internal control policies.
• Establish CI/CD pipelines, unit testing frameworks, and engineering best practices to support high-availability platform deployments.

Skills

Python

PySpark

Databricks

Kafka

FastAPI

GenAI Agents

Google ADK

LLMs

MCP

Kubernetes

OpenShift

CI/CD

Data Modeling

Data Warehousing

Data Mesh

Tools

Starburst

OpenShift

Docker

Kubernetes

GitHub Copilot

Devin.AI

Job description

Roles and Responsibilities

• Design, develop, and maintain scalable, enterprise-grade AI agents and high-volume ELT/ETL pipelines using Python, PySpark, Databricks, Kafka, and FastAPI.


• Implement and deploy GenAI agents using Google ADK and Google Flash 2.5+ LLMs to power application automation and workflow optimization via Human-in-the-Loop (HIL) design.


• Build and maintain data federation layers for Lambda and Data Mesh architectures using Starburst to enable machine learning, deep learning, and NLP use cases.


• Develop, deploy, and automate resilient microservice integrations supporting data-intensive applications on Kubernetes and OpenShift cloud-native platforms.


• Integrate agentic AI tools such as Devin.AI and GitHub Copilot via Model Context Protocol (MCP) and advanced prompt engineering to maximize development velocity.


• Enforce data quality, security, and risk compliance across the data lifecycle, ensuring adherence to regulatory standards and internal control policies.


• Establish CI/CD pipelines, unit testing frameworks, and engineering best practices to support high-availability platform deployments.

Preferred Candidate Profile

Experience:

• Overall Experience: 8+ years in large-scale software engineering or data platform development.


• Relevant Experience: 5+ years of hands-on technical lead experience in Python, PySpark, and Databricks pipeline architectures.


• Domain Experience: Background in Banking, Financial Services, Retail Products (Cards, Mortgages, Deposits), or Wealth Management risk domains is strongly preferred.

Technical Expertise

• Core Data Engineering: Python, PySpark, Databricks, SQL, Kafka, Data Modeling, Data Warehousing.


• AI & Microservices: GenAI Agents, Google ADK, LLM Integrations (Flash 2.5+), MCP, FastAPI, Microservices, Prompt Engineering.


• Platform & Infrastructure: Data Mesh, Starburst, OpenShift, Kubernetes, Docker, CI/CD.