Data Engineer

NITYO; Mumbai; On-site; INR 1,500,000 - 2,100,000; Full time

Job summary

NITYO is seeking a Data Engineer in Mumbai to design and manage data pipelines, vector infrastructure, and real-time processing systems. You will work with distributed data platforms and orchestration tools to ensure robust data processing and semantic consistency.

The role emphasizes building and maintaining vector indexing with pgvector, Redis, and Azure AI Search, while implementing privacy controls and zero-trust access within AI pipelines.

Qualifications

• Experience with distributed data systems, SQL, and orchestration tools.
• Experience tuning high-throughput database infrastructure.
• Knowledge of Google's GECX is a plus.

Responsibilities

• Build ETL flows for structured/unstructured data, ensuring normalization, deduplication, and semantic consistency.
• Manage pgvector, Azure AI Search, Redis vector indexing, and hybrid search layers.
• Implement zero-trust access, privacy controls, and compliance within AI context pipelines.
• Build event-driven architectures for real-time processing and refresh embeddings and indexes.

Skills

ETL & Data Modeling

Vector Databases

Distributed Data Systems

Data Governance

Real-time Embedding Updates

Chunking & Embeddings

Search Infrastructure

Performance Tuning

Data Quality & Lineage

Job description

Data Engineer

This Data Engineer position involves building and managing data pipelines, vector infrastructure, and real-time processing architectures. The ideal candidate will have experience with distributed data systems, SQL, and orchestration tools.

Key Responsibilities

• Build ETL flows for structured/unstructured data, ensuring normalization, deduplication, and semantic consistency.
• Manage pgvector, Azure AI Search, Redis vector indexing, and hybrid search layers.
• Implement zero-trust access, privacy controls, and compliance within AI context pipelines.
• Build event-driven architectures for real-time processing and continuously refresh embeddings and indexes.

Requirements

• Deep experience with distributed data systems, SQL, and orchestration tools.
• Experience tuning high-throughput database infrastructure.
• Knowledge of Google's GECX is a plus.
• Familiarity with chunking strategies and embedding models.

Work Arrangements

• Location: Mumbai, Maharashtra, India
• Type: Contract
• Remote: Not specified

Skills

• ETL & Data Modeling: Designing pipelines for structured/unstructured data, normalization, deduplication, and semantic consistency.
• Vector Databases: pgvector, Redis, Azure AI Search, hybrid search, and index optimization.
• Distributed Data Systems: Kafka, Spark, Flink, or similar event-driven architectures.
• Data Governance: Zero-trust access, privacy controls, compliance, and auditability.
• Real-time Embedding Updates: Event-driven refresh pipelines for RAG and agent memory systems.
• Chunking & Embeddings: Semantic chunking, metadata tagging, and embedding model selection.
• Search Infrastructure: BM25, hybrid search, inverted indexes, and ranking algorithms.
• Performance Tuning: High-throughput read/write optimization.
• Data Quality & Lineage: Validation, schema enforcement, and lineage tracking.