Senior Data Engineer – Knowledge Graph & AI Platform

AiFA Labs; Hyderabad; Hybrid; INR 7,194,244 - 9,892,086; Full time

Benefits offered by this job

Competitive compensation with equity options

Flexible remote/hybrid work setup

Learning budget and conference support

Job summary

A leading technology firm is seeking a Senior Data Engineer to build and maintain its core data infrastructure for an enterprise AI platform. The candidate will design and implement scalable data pipelines, develop knowledge graphs, and ensure data quality. Required skills include extensive data engineering experience, proficiency in Python and MongoDB, as well as expertise with vector databases. The position offers competitive compensation, flexible working options, and opportunities to work with cutting-edge technologies.

Qualifications

• 5+ years of experience with production-grade data engineering pipelines.
• Expertise in Python for writing clean, testable, maintainable code.
• Proficient with MongoDB for schema design and performance tuning.
• Experience with vector databases like Qdrant or Weaviate.
• Good knowledge of SQL for complex queries and optimization.

Responsibilities

• Build and maintain core data infrastructure for an AI platform.
• Develop scalable data pipelines and knowledge graphs.
• Integrate enterprise data and prepare for LLM consumption.
• Ensure data quality through validation and lineage tracking.

Skills

Data Engineering experience

Python

MongoDB

Vector databases

Document processing

SQL

ETL/ELT at scale

Pipeline orchestration tools

Tools

Airflow

Kafka

Apache Spark

Job description

Location: Remote / Hybrid (India)

Employment Type: Full-Time

Reporting To: Platform Architect

Role Overview

The Senior Data Engineer will build and maintain the core data infrastructure for an enterprise AI platform. This role focuses on designing scalable data pipelines, developing knowledge graphs, and preparing structured and unstructured data for AI and LLM-based applications.

Roles & Responsibilities

Data Pipeline Development

• Design and build scalable data ingestion pipelines from enterprise systems (ERP, documentation tools, version control, and project management tools)
• Develop connectors for structured, semi-structured, and unstructured data
• Implement incremental data loads, change data capture (CDC), and real-time sync
• Ensure data quality through validation, deduplication, and lineage tracking

Knowledge Graph Engineering

• Design ontologies and graph schemas for complex enterprise relationships
• Implement entity resolution and relationship inference across data sources
• Build APIs and query interfaces for graph traversal
• Optimize graph storage and query performance for large-scale usage

Enterprise Data Integration

• Extract and model enterprise metadata such as business rules and data dictionaries
• Parse and semantically index documents and code artifacts
• Build integrations with enterprise APIs and internal platforms

AI & LLM Data Infrastructure

• Prepare structured and contextual data for LLM consumption
• Design embedding strategies and manage vector databases for semantic search
• Build memory and context management systems for stateful AI applications

Required Skills

Core Requirements

• 5+ years of Data Engineering experience with production-grade pipelines
• Strong Python skills (clean, testable, maintainable code)
• MongoDB expertise (schema design, aggregation pipelines, indexing, performance tuning)
• Vector databases experience (Qdrant, Pinecone, Weaviate, pgvector)
• Document processing experience (chunking, metadata extraction, PDFs/Word/HTML; LangChain or similar)

Strong SQL Skills

• Complex queries, joins, window functions, optimization

ETL/ELT at scale

• Incremental loads, CDC, idempotent pipelines

Pipeline orchestration tools

• Airflow, Dagster, Prefect, or similar

Good to Have / Strong Plus

• Experience building production RAG pipelines
• Deep understanding of embedding models and dimensionality
• Graph databases (Neo4j) and Cypher query expertise
• LLM application development using LangChain or Lang Graph
• Streaming systems (Kafka, Flink) for real-time pipelines
• Hybrid search (vector + keyword/metadata filtering)
• Apache Spark for large-scale transformations

What We Offer

• Work on cutting-edge AI and knowledge graph technologies
• Build foundational infrastructure for an enterprise AI platform
• Competitive compensation with equity options
• Flexible remote/hybrid work setup
• Learning budget and conference support