Senior / Lead Data Engineer – Service Layer / QTC Integration Platform

TALPRO INDIA PRIVATE LIMITED; Bengaluru; On-site; INR 2,400,000 - 4,200,000; Full time

Job summary

Talpro is seeking a Senior/Lead Data Engineer in Bengaluru to design, build, and operate the Service Layer powering QTC, data pipelines, and enterprise integrations. You will lead a team, own key delivery, and set standards for reliability, security, and performance across distributed systems.

The role demands hands-on leadership, deep cloud-native expertise on AWS, Kafka, and data platforms, with a focus on scalability, observability, and enterprise-grade interfaces.

Qualifications

• Bachelor's degree in Computer Science, Engineering, Information Technology or related field.
• 8+ years of software engineering experience.
• 4+ years focused on cloud-native, data engineering, or distributed platforms.

Responsibilities

• Own end-to-end technical delivery of key initiatives with quality, scalability, security, and operability.
• Lead and mentor a team of engineers through planning, guidance, code/design reviews and delivery.
• Drive engineering best practices across CI/CD, observability, resiliency, security, and cloud-native patterns.
• Manage multiple initiatives, balancing business priorities, tech debt, and platform evolution.
• Design decisions for event-driven integrations, canonical data models, and platform capabilities.

Skills

Go / Golang

Python

AWS

Kafka

Event-driven architecture

Distributed systems

Leadership

Education

Bachelor's degree in Computer Science/Engineering

Tools

AWS

Kafka

Snowflake

PostgreSQL

S3

Lambda

API Gateway

Job description

Job Description

Talpro is leading the way in transforming the talent acquisition landscape. We deliver innovative, sustainable, and cost-effective recruitment solutions tailored to today’s business needs. Our mission is to offer comprehensive hiring strategies that address immediate recruitment demands while laying the groundwork for long‑term success.

Role Overview

We are looking for a strong, hands‑on, and highly autonomous Senior / Lead Data Engineer to design, build, and operate the Service Layer platform powering enterprise Quote‑to‑Cash (QTC), customer lifecycle, finance, and enterprise application integrations.

This role requires deep expertise in distributed systems, event‑driven architecture, microservices, data engineering, cloud‑native development, and platform reliability. The candidate will also provide technical leadership to a team of engineers, driving solution design, engineering standards, delivery execution, and operational support across multiple integration initiatives.

The platform is built using AWS, Kafka / AWS MSK, Go, Python, canonical data models, asynchronous messaging, and cloud‑native architecture patterns, processing high‑volume, business‑critical events with strong focus on reliability, security, scalability, observability, and operational readiness.

Key Responsibilities

Technical Delivery & Engineering Leadership

• Own end‑to‑end technical delivery of key initiatives, ensuring solutions are delivered with quality, scalability, security, and operational readiness.
• Lead and mentor a team of engineers through task planning, technical guidance, code reviews, design reviews, and delivery management.
• Drive engineering best practices across software development, CI/CD, observability, resiliency, security, and cloud‑native architecture.
• Manage multiple concurrent initiatives while balancing business priorities, technical debt, and platform evolution.
• Drive architecture and design decisions for event‑driven integrations, canonical data models, orchestration frameworks, and platform capabilities.
• Define and enforce engineering standards, secure‑by‑design principles, cloud best practices, data governance, and non‑functional requirements.
• Design and build scalable Service Layer foundation capabilities including messaging, orchestration, observability, reconciliation, data migration, operational tooling, and canonical data management.
• Ensure platform design supports resilience, scalability, fault tolerance, performance, and supportability.

Event‑Driven Integration & Microservices

• Design, develop, and operate real‑time event‑driven integration services using Go / Golang and Python.
• Build and enhance the Service Layer platform to orchestrate Quote‑to‑Cash workflows across enterprise applications.
• Design and implement scalable Kafka / AWS MSK‑based event processing with support for:
  • Retries
  • Idempotency
  • Sequencing
  • Message replay
  • Fault recovery
  • Dead‑letter queues
  • Eventual consistency
• Build secure and scalable RESTful APIs, integration services, and enterprise system interfaces.

Data Engineering, Migration & Reconciliation

• Design and implement large‑scale ETL, migration, reconciliation, and data‑quality solutions using:
  • AWS Glue
  • Step Functions
  • Snowflake
  • PostgreSQL
  • ClickHouse
• Build and enforce canonical data models, data contracts, transformation layers, and validation frameworks.
• Support enterprise integrations by standardizing communication across multiple business systems.
• Ensure data accuracy, traceability, consistency, and operational reliability across the platform.
• Design and build cloud‑native applications on AWS using:
  • AWS Lambda
  • API Gateway
  • S3
  • RDS / Aurora PostgreSQL
  • DynamoDB
  • SQS
  • SNS
  • EventBridge
  • IAM
  • CloudWatch
  • VPC
• Build high‑availability, business‑critical applications with strong focus on performance, security, and operational supportability.

Observability, Monitoring & Operations

• Design and implement observability, monitoring, alerting, and operational dashboards for end‑to‑end workflow visibility.
• Work with tools such as:
  • CloudWatch
  • Grafana
  • Datadog
  • Prometheus
  • OpenTelemetry
• Support production operations including incident management, root‑cause analysis, troubleshooting, and performance optimization.
• Drive metrics‑driven improvements for reliability, scalability, and platform performance.

Collaboration & Stakeholder Management

• Partner with Product Managers, Architects, Enterprise Data teams, Engineering, DevOps, and business stakeholders.
• Define technical roadmaps, execution plans, dependencies, and delivery milestones.
• Communicate technical risks, trade‑offs, design decisions, and mitigation plans clearly to stakeholders.
• Collaborate across multiple teams to ensure integration initiatives are delivered efficiently.

Must‑Have Skills

Leadership & Delivery

• Proven experience leading and mentoring engineering teams.
• Strong experience in task planning, technical guidance, code reviews, and delivery management.
• Ability to drive architecture decisions for large‑scale distributed systems and enterprise integration platforms.
• Experience managing multiple parallel technical initiatives.
• Strong hands‑on experience in:
  • Go / Golang
  • Python
• Experience building scalable backend services, event‑driven integrations, orchestration workflows, and data processing solutions.
• Strong understanding of REST API development and secure API design.
• Strong experience designing and building cloud‑native applications on AWS.
• Hands‑on experience with:
  • Lambda
  • API Gateway
  • S3
  • RDS / Aurora PostgreSQL
  • DynamoDB
  • SQS / SNS
  • EventBridge
  • IAM
  • CloudWatch
  • VPC
• Strong understanding of AWS serverless architecture patterns.
• Strong experience designing and operating event‑driven and asynchronous systems using:
  • Kafka
  • AWS MSK
  • RabbitMQ or equivalent messaging technologies
• Strong understanding of:
  • Fault tolerance
  • Idempotency
  • Retry patterns
  • Dead‑letter queues
  • Message replay
  • Eventual consistency
  • Sequencing
  • Event orchestration
• Strong experience with:
  • PostgreSQL
  • ClickHouse
  • Relational databases
  • DynamoDB / NoSQL databases
• Strong knowledge of:
  • Data modelling
  • Query optimization
  • Performance tuning
  • Data validation
  • Reconciliation frameworks
• Experience with AWS Glue, Step Functions, Snowflake, and large‑scale data workflows.

CI/CD, Security & Observability

• Experience with CI/CD pipelines, automated testing, code quality, and secure delivery using tools such as:
  • GitLab CI
  • Jenkins
  • SonarQube
  • Trivy
  • Semgrep
• Strong understanding of cloud security and secure‑by‑design principles including:
  • IAM
  • OAuth2
  • JWT authentication
  • Encryption at rest and in transit
  • Secrets Manager
  • Vulnerability management
  • Audit logging
  • Compliance controls

Required Qualifications

• Bachelor’s degree in Computer Science, Engineering, Information Technology, or related discipline.
• 8+ years of software engineering experience.
• At least 4+ years focused on cloud‑native, data engineering, integration, or distributed platforms.
• 4+ years building cloud‑native solutions on AWS.
• 4+ years working with Go / Golang, Python, AWS services, and Kafka in production environments.
• Proven experience designing, building, and operating highly available enterprise integration and event‑driven platforms.
• Strong experience in production support, incident management, root‑cause analysis, and performance optimization.

Nice to Have

• AWS certifications such as:
  • AWS Certified Data Engineer – Associate
  • AWS Certified Solutions Architect – Associate
  • AWS Certified Developer – Associate
  • Equivalent AWS certifications
• Experience in:
  • Quote‑to‑Cash
  • CRM
  • ERP
  • Customer Lifecycle Management
  • Enterprise Data domains
• Exposure to large‑scale finance, billing, or enterprise integration transformation programmes.
• Experience building operational dashboards and platform‑level reporting for engineering and business teams.