Data Engineer

Straive; Pune District; On-site; INR 4,000,000 - 7,000,000; Full time

Job summary

Straive is seeking an experienced Data Engineering lead to architect and implement enterprise ELT/ETL pipelines using Python, PySpark, Kafka, and Databricks for risk data at scale. You will design GenAI agents with Google ADK and LLMs, and build microservice integrations on OpenShift/Kubernetes with robust CI/CD.

Ideal candidates have 8+ years in large-scale app development and 5+ years in a Python/PySpark leadership role, with a CS/Engineering degree (Master's preferred).

Qualifications

• 8+ years in large-scale application development.
• 5+ years leading Python/PySpark data engineering teams.
• Bachelor's degree in CS/Engineering; Master's preferred.

Responsibilities

• Architect and maintain ELT/ETL pipelines with Python, PySpark, Kafka, and Databricks for large-scale risk data.
• Build GenAI agents using Google ADK, LLMs, MCP, and Human-in-the-Loop workflows.
• Design and deploy microservice integrations on OpenShift/Kubernetes with CI/CD pipelines.
• Implement data federation with Lambda/Data Mesh via Starburst for AI/ML use cases.
• Leverage AI tooling like Devin.AI and GitHub Copilot to accelerate development velocity.
• Enforce data governance, risk management, and regulatory compliance across platforms.

Skills

Python

PySpark

Databricks

Kafka

Kubernetes

OpenShift

LLMs

FastAPI

Spring Boot

Microservices

SQL

Starburst

Data Mesh

Google ADK

GitHub Copilot

Education

Bachelor's degree in Computer Science or Engineering

Master's degree preferred

Tools

Databricks

OpenShift

Kubernetes

Docker

Google ADK

LLMs

Kafka

Starburst

SQL

Job description

Roles and Responsibilities

• Architect and maintain enterprise-grade ELT and ETL data pipelines using Python, PySpark, Kafka, and Databricks to manage large-scale risk data.
• Build and deploy GenAI agents utilizing Google ADK, Google Flash 2.5+ LLMs, and Model Context Protocol (MCP) integrated with Human-in-the-Loop workflows.
• Design, automate, and deploy microservice integrations for data-intensive applications on OpenShift and Kubernetes using robust CI/CD pipelines.
• Implement data federation layers supporting Lambda and Data Mesh architectures via Starburst to enable AI/ML and NLP use cases.
• Leverage agentic AI platforms and development assistants such as Devin.AI and GitHub Copilot with prompt engineering to increase engineering velocity.
• Enforce data governance, risk management policies, and regulatory compliance standards across all data platforms.

Preferred Candidate Profile

• Work Experience: 8+ years in large-scale application development with 5+ years in a Python and PySpark Data Engineering lead role.
• Educational Background: Bachelor's degree in Computer Science, Engineering, or a related field (Master's degree preferred).
• Core Technical Skills: Python, PySpark, Databricks, Google ADK, LLMs, FastAPI, Spring Boot, Microservices, Kafka, SQL, Data Mesh, Starburst.
• Infrastructure and Cloud: Kubernetes, OpenShift, Docker, Cloud-Native Infrastructure, CI/CD pipelines.
• Industry Context: Data engineering experience in Banking Risk, Retail Products, Cards, Mortgage, Deposits, or Wealth Management.
• Assumed Requirements / Certifications: Databricks Certified Data Engineer, AWS Certified Data Analytics, or Azure Data Engineer Associate.