Senior AI Data Engineer — AI Harness & Iceberg Platform

Ncs; Singapore; On-site; SGD 120,000 - 180,000; Full time

Job summary

NCS is seeking an autonomous AI Engineer to design and build the AI harness, agent framework, and reusable components for a modern data platform based on Apache Iceberg, Trino and Parquet. You will enable multiple specialized agents to collaborate, execute tasks, maintain state, orchestrate workflows, and deliver end-to-end data engineering capabilities.

Depending on experience, focus on task execution, independent delivery, or technical leadership.

Qualifications

• Experience with Generative AI, Agentic AI, or LLM-based solutions.
• Proficiency in Python, including FastAPI, Pydantic and Pandas.
• Experience designing and implementing AI Harness or multi-agent frameworks.
• Hands-on experience with LangChain & LangGraph.
• Experience with Apache Iceberg, Trino, and Parquet-based lakehouse architectures.
• Experience building data ingestion, transformation, and orchestration pipelines.
• Strong knowledge of metadata management, data lineage, and data governance practices.
• Experience developing REST APIs, SDKs, and reusable engineering frameworks.
• Experience with observability, monitoring, tracing, and operational support practices.
• Strong understanding of distributed systems, event-driven architectures, and microservices patterns.
• Proven ability to design, develop, and operationalize enterprise-grade AI agents for data discovery, ingestion, transformation, optimization, governance, and observability.
• Familiar with modern data engineering technologies, including DuckDB, dbt, and related engineering tools for data transformation, modelling, and pipeline development.
• Senior engineers will be expected to have demonstrated evidence of leading technical designs, guiding junior engineering team members, working independently with clients and delivery teams, and translating AI/data requirements into scalable, secure, and deployable solutions.

Responsibilities

• AI Harness Development and agent orchestration.
• Workflow execution, state management, and memory handling.
• Tool and function calling, agent-to-agent communication.
• Human approval workflows, logging, and audit trails.
• Security, access controls, observability and monitoring.
• Specialized agents to discover Iceberg catalogs, profile datasets, generate metadata and lineage.
• Create Parquet datasets, register Iceberg tables, generate Trino SQL.
• Build transformation workflows and curated datasets.
• Validate schemas, data integrity, and detect anomalies.
• Monitor pipelines and agent activities; generate alerts and insights.

Skills

Python

FastAPI

Pydantic

Pandas

LangChain

LangGraph

Apache Iceberg

Trino

Parquet

Data pipelines

LLM-based solutions

REST APIs

Distributed systems

Data governance

Observability

Tools

DuckDB

dbt

Iceberg tooling

Trino tooling

Job description

NCS is seeking an autonomous AI Engineer to design and build the AI harness, agent framework, and reusable components for a modern data platform based on Apache Iceberg, Trino and Parquet. You will enable multiple specialized agents to collaborate, execute tasks, maintain state, orchestrate workflows, and deliver end-to-end data engineering capabilities.

Depending on experience, focus on task execution, independent delivery, or technical leadership.