Senior AI Platform & Data Engineer

amIT Global Solutions Sdn Bhd; Indonesia; On-site; IDR 400,000,000 - 700,000,000; Full time

Job summary

Tata Consultancy Services (TCS) seeks a Senior AI Platform & Data Engineer to build and govern a universal data platform, enabling AI agents, LLMs, and retrieval-based workflows. You will integrate varied data sources, ensure clean data, and drive GenAI experiments across UDP ingestion, governance, and semantic search capabilities.

You will implement pipeline observability, data quality systems, and knowledge bases, while enabling scalable AI agent consumption through structured metadata and

Qualifications

• Bachelor's degree in CS/DS/IT or related field.
• 4+ years as Data/AI/Analytics engineer in data-driven environments.
• Proficient in Python and SQL with data quality tooling.
• Experience with data profiling, testing, and anomaly detection.

Responsibilities

• Integrate UDP ingestion pipelines to unify data ecosystems.
• Lead GenAI, LLM, and knowledge-base initiatives.
• Implement data pipeline monitoring platforms for health and uptime.
• Set up automated data anomaly detection for schema drift and nulls.
• Build data catalogs and metadata schemas for self-serve data.
• Prepare data for AI agent consumption and semantic retrieval.
• Design and implement robust RAG pipelines and vector search.

Skills

Python

SQL

Data engineering

AI engineering

Analytics engineering

Big data

Data quality

Education

Bachelor's degree in Computer Science / Data Science / IT

Tools

Great Expectations

Soda

dbt tests

Monte Carlo

Datadog

Prometheus

Elasticsearch

Weaviate

Qdrant

LangChain

LangGraph

Dify

LlamaIndex

Vertex AI

BigQuery

GKE

Docker

Kubernetes

Streamlit

Metabase

PowerBI

Job description

About TCS:

A purpose-led organization that is building a meaningful future through innovation, technology, and collective knowledge. We're #BuildingOnBelief.

Tata Consultancy Services (TCS) is a global leader in IT services, digital and business solutions that partners with its clients to simplify, strengthen and transform their businesses. TCS offers a consulting-led, integrated portfolio of IT, BPS, infrastructure, engineering and assurance services. We ensure the highest levels of certainty and satisfaction through a deep-set commitment to our clients, comprehensive industry expertise and a global network of innovation and delivery centers. For more information, visit us at www.tcs.com.

Role Overview

We are seeking a highly skilled and versatile Senior AI Platform & Data Engineer to join our team via Managed Services. In this role, you will act as a critical builder of our Universal Data Platform (UDP), bridging the gap between robust Data Engineering, Analytics Engineering, and Generative AI enablement.

Your primary objective is to integrate diverse enterprise data sources and build reliable, automated data pipelines that feed a structured, self-service data catalog and knowledge base. You will ensure that company-wide data is not only clean and well-governed but also strictly monitored for anomalies (such as missing null values or schema drift) so that it remains optimized for agentification—enabling AI Agents, Large Language Models (LLMs), and Retrieval-Augmented Generation (RAG) workflows to securely and seamlessly consume trusted enterprise knowledge.

Required Qualifications & Skills:

• Bachelor's Degree in Computer Science, Data Science, Information Technology, Computer Engineering, Software Engineering, Artificial Intelligence, or a related field.
• Minimum 6-10 years of experience in Data Engineering, AI Engineering, Machine Learning Engineering, Analytics Engineering, or related fields.
• Experience: 4+ years of experience as a Data Engineer, AI Engineer, or Analytics Engineer, ideally within environments emphasizing big data integration, platform enablement, or AI product development.
• Programming Languages: Exceptional proficiency in Python and SQL.
• Data Quality & Monitoring Tools:
• Practical experience with data profiling, testing, and anomaly detection frameworks (e.g., Great Expectations, Soda, dbt tests).
• Familiarity with data observability and pipeline monitoring suites to track data completeness (null checks, range boundaries, type assertions).

Key Responsibilities

• Universal Data Platform Integration: Collaborate on the design and execution of UDP ingestion pipelines, integrating various legacy and modern source systems to establish a unified data ecosystem.
• Generative AI & LLM Implementation: Lead experimentation and development of GenAI solutions, including Knowledge Bases, Chatbots (e.g., Copilots), text-to-SQL workflows, and document summarization using state-of-the-art frameworks.
• Pipeline Observability & Monitoring: Implement and manage specialized data pipeline monitoring platforms (e.g., Monte Carlo, Datadog, Prometheus, Soda) to track pipeline health, execution times, failures, and maintain high platform uptime.
• Data Quality & Anomaly Detection: Set up automated data anomaly detection systems to identify data discrepancies, run-time schema drift, missing metadata, and critical data gaps (such as unexpected null values or zero-records).
• Knowledge Base & Data Cataloging: Build and maintain proper data catalogs, metadata schemas, and centralized knowledge bases, enabling anyone in the company to easily discover, manage, and own their business domain's data.
• Enablement for Agentification: Structure and optimize corporate data assets specifically for AI agent consumption. This includes establishing proper indexing, document chunking pipelines, and preparing unstructured/structured data for high-performance semantic retrieval.
• Generative AI & Semantic Search Engineering: Design and implement robust RAG pipelines, semantic search layers, and vector databases (e.g., Elasticsearch, Weaviate, Qdrant) to connect the enterprise data platform with downstream LLMs.

Generative AI & Search Tech Stack:

• Solid understanding of vector search concepts and databases (Weaviate, Qdrant, Elasticsearch, or Vertex AI Vector Search).
• Experience with LLM orchestration, RAG pipelines, and agent frameworks (LangChain, LangGraph, Dify, LlamaIndex).

Cloud & Platform Infrastructure:

• Experience with Google Cloud Platform (GCP) services (BigQuery, Vertex AI, GKE, CloudRun).
• Strong understanding of containerization (Docker, Kubernetes) and deploying data/AI applications as scalable APIs.
• Data Visualization & Prototyping: Ability to quickly spin up interactive dashboards or internal portals (Streamlit, Metabase, PowerBI) to demonstrate data cataloging, quality metrics, and agent capabilities to stakeholders.

Ideal Candidate Profile

• Data Quality Champion: You treat data quality as a product priority, knowing that AI agents and downstream applications are only as good as the underlying data inputs (preventing Garbage In, Garbage Out).
• Platform Thinker: You design reusable, modular pipelines, templates, and metadata schemas that empower the entire organization to easily discover and self-serve data.
• AI-Native Mindset: You understand how LLMs and agents think and actively structure enterprise knowledge bases to be easily parsed and queried by AI systems.
• Collaboration Champion: You can work closely with business units to help them organize, catalog, monitor, and truly own their domain's data.