Senior AI Platform & Data Engineer for GenAI Pipelines

Tata Consultancy Services; Jakarta Pusat; On-site; IDR 600,000,000 - 1,000,000,000; Full time

Job summary

Tata Consultancy Services (TCS) seeks a Senior AI Platform & Data Engineer to join via Managed Services, building the Universal Data Platform and enabling AI agents, LLMs, and RAG workflows. You will integrate diverse data sources, automate pipelines, and ensure clean, governed data for enterprise use.

You will lead GenAI experiments, implement semantic search, vector stores, and monitoring platforms, delivering scalable data solutions and trusted knowledge assets for AI applications.

Qualifications

• Bachelor's Degree in Computer Science, Data Science, Information Technology, Computer Engineering, Software Engineering, Artificial Intelligence, or a related field.
• Minimum 6-10 years of experience in Data Engineering, AI Engineering, Machine Learning Engineering, Analytics Engineering, or related fields.
• 4+ years of experience as a Data Engineer, AI Engineer, or Analytics Engineer, ideally within environments emphasizing big data integration, platform enablement, or AI product development.
• Programming Languages: Python and SQL.
• Practical experience with data profiling, testing, and anomaly detection frameworks (e.g., Great Expectations, Soda, dbt tests).
• Familiarity with data observability and pipeline monitoring suites to track data completeness (null checks, range boundaries, type assertions).

Responsibilities

• Universal Data Platform Integration: Collaborate on the design and execution of UDP ingestion pipelines, integrating various legacy and modern source systems to establish a unified data ecosystem.
• Generative AI & LLM Implementation: Lead experimentation and development of GenAI solutions, including Knowledge Bases, Chatbots (e.g., Copilots), text-to-SQL workflows, and document summarization using state-of-the-art frameworks.
• Pipeline Observability & Monitoring: Implement and manage specialized data pipeline monitoring platforms (e.g., Monte Carlo, Datadog, Prometheus, Soda) to track pipeline health, execution times, failures, and maintain high platform uptime.
• Data Quality & Anomaly Detection: Set up automated data anomaly detection systems to identify data discrepancies, run-time schema drift, missing metadata, and critical data gaps (such as unexpected null values or zero-records).
• Knowledge Base & Data Cataloging: Build and maintain proper data catalogs, metadata schemas, and centralized knowledge bases, enabling anyone in the company to easily discover, manage, and own their business domain's data.
• Enablement for Agentification: Structure and optimize corporate data assets specifically for AI agent consumption. This includes establishing proper indexing, document chunking pipelines, and preparing unstructured/structured data for high-performance semantic retrieval.
• Generative AI & Semantic Search Engineering: Design and implement robust RAG pipelines, semantic search layers, and vector databases (e.g., Elasticsearch, Weaviate, Qdrant) to connect the enterprise data platform with downstream LLMs.
• Solid understanding of vector search concepts and databases (Weaviate, Qdrant, Elasticsearch, or Vertex AI Vector Search).
• Experience with LLM orchestration, RAG pipelines, and agent frameworks (LangChain, LangGraph, Dify, LlamaIndex).
• Experience with Google Cloud Platform (GCP) services (BigQuery, Vertex AI, GKE, CloudRun).
• Strong understanding of containerization (Docker, Kubernetes) and deploying data/AI applications as scalable APIs.
• Data Visualization & Prototyping: Ability to quickly spin up interactive dashboards or internal portals (Streamlit, Metabase, PowerBI) to demonstrate data cataloging, quality metrics, and agent capabilities to stakeholders.

Skills

Python

SQL

Data engineering

AI engineering

Analytics engineering

Big data

Education

Bachelor's degree

Tools

Great Expectations

Soda

dbt tests

Monte Carlo

Datadog

Prometheus

Elasticsearch

Weaviate

Qdrant

LangChain

Job description

Tata Consultancy Services (TCS) seeks a Senior AI Platform & Data Engineer to join via Managed Services, building the Universal Data Platform and enabling AI agents, LLMs, and RAG workflows. You will integrate diverse data sources, automate pipelines, and ensure clean, governed data for enterprise use.

You will lead GenAI experiments, implement semantic search, vector stores, and monitoring platforms, delivering scalable data solutions and trusted knowledge assets for AI applications.