Data Platform Engineering Lead - AITDS

Cognizant; Hyderabad; On-site; INR 4,000,000 - 6,400,000; Full time

Job summary

Cognizant is hiring a Data Platform Engineering Lead in Hyderabad to build AI-ready, scalable data platforms. You will design scalable pipelines, lakehouse architectures, and governed data foundations for AI/ML and LLM-driven applications.

The role covers backend services, feature stores, and data products, with emphasis on CI/CD, containers, and modern software engineering practices. You will mentor engineers and collaborate across cloud infra teams.

Qualifications

• 8+ years in data engineering and backend development.
• Strong Python, APIs, and microservices experience.
• Deep experience in Spark/PySpark or equivalent.
• Expertise in ETL/ELT pipelines and lakehouse architectures.
• Experience with feature stores and AI-ready datasets.
• Experience with Azure/AWS/GCP.
• Familiarity with Docker, Kubernetes, CI/CD.
• Familiarity with LLM/GenAI integration patterns.
• Familiarity with metadata and governance tooling.
• Strong system design and problem-solving skills.

Responsibilities

• Build & Deploy Data Platform Solutions.
• Design and develop data pipelines for batch and streaming workloads using Python and Spark/PySpark.
• Build and manage AI-ready datasets, feature pipelines, and training data foundations.
• Develop backend services and APIs to expose data products.
• Implement microservices and event-driven architectures for data ingestion, processing, and serving.
• Ensure the platform is scalable, performant, and maintainable.
• Mentor engineers on data engineering patterns and best practices.
• Enable AI/ML & LLM Integration.
• Develop AI-ready pipelines for model training, evaluation, and inference.
• Enable feature store capabilities and reproducible datasets.
• Support integration with LLM/AI services (RAG, embeddings, inference APIs).
• Oversee implementation on cloud infrastructure with Azure/AWS/GCP.
• Collaborate with infra teams on cloud deployments.
• Build and integrate data lakes, lake-houses, and SQL/NoSQL systems.
• Enable data platform integration with AI/ML systems.
• Implement containerized and serverless architectures.
• Implement CI/CD pipelines and observability.
• Define and enforce data quality frameworks.
• Support metadata, lineage, and governance.
• Optimize platform performance, reliability, and scalability.

Skills

Data engineering

Python

APIs

Microservices

Spark/PySpark

ETL/ELT

Lakehouse

Feature stores

Cloud platforms

Docker

Kubernetes

CI/CD

LLM integration

Data governance

System design

Tools

Docker

Kubernetes

CI/CD

Job description

Job Description

Data Platform Engineering Lead

Level: M / SA

Role Overview

A hands-on technical lead responsible for building AI-ready enterprise data platforms, including scalable pipelines, lakehouse architectures, and governed data foundations that enable model training, evaluation, and inference. The role extends to backend services and integration layers that expose curated datasets, feature stores, and data products for AI/ML and LLM-driven applications.

Key Responsibilities

• Build & Deploy Data Platform Solutions
• Design and develop data pipelines for batch and streaming workloads using Python and Spark/PySpark
• Build and manage AI-ready datasets, feature pipelines, and training data foundations
• Develop backend services and APIs (Python – FastAPI or equivalent) to expose data products
• Implement microservices and event-driven architectures for data ingestion, processing, and serving
• Ensure the platform is scalable, performant, and maintainable
• Mentor engineers on data engineering patterns and best practices.
• Enable AI/ML & LLM Integration
• Develop AI-ready pipelines for model training, evaluation, and inference
• Enable feature store capabilities and reproducible datasets
• Support integration with LLM/AI services (RAG, embeddings, inference APIs)
• Enable data-to-AI pipelines including vectorization and retrieval workflows.
• Oversee Implementation on Cloud Infrastructure
• Collaborate with infra teams on Azure, AWS, or GCP
• Build and integrate data lakes, lake-houses, SQL/NoSQL systems
• Enable integration between data platforms and AI/ML systems
• Implement containerized and serverless architectures.
• Implement Modern Software Engineering Practices
• Implement CI/CD pipelines and observability
• Define and enforce data quality frameworks
• Support metadata, lineage, and governance
• Optimize platform performance, reliability, and scalability.

Required Capabilities / Skills / Experience

• 8+ years in data engineering and backend development
• Strong Python, APIs, and microservices experience
• Deep experience in Spark/PySpark or equivalent
• Expertise in ETL/ELT pipelines and lakehouse architectures
• Experience with feature stores and AI-ready datasets
• Experience with Azure/AWS/GCP
• Familiarity with Docker, Kubernetes, CI/CD
• Familiarity with LLM/GenAI integration patterns
• Familiarity with metadata and governance tooling
• Strong system design and problem-solving skills.