Senior Full Stack Data Platform Engineer

Millennium Consulting; Bengaluru; On-site; INR 5,000,000 - 9,000,000; Full time

Job summary

Millennium Consulting is seeking a Senior Full Stack Data Platform Engineer to architect high-throughput data platforms using Python, Java and C++. You will build and optimize a config-driven, plugin-enabled system to ingest gigabytes of unstructured data and enable rich data consumption through robust APIs.

You will bridge ML research and real-time trading decisions in a poly-language environment, delivering low-latency pipelines, NLP model serving, and scalable backend services across

Qualifications

• 5+ years of software engineering experience, with a focus on data platforms.
• Proficiency in Python and Java or C++, with ability to switch contexts.
• Experience building data pipelines, ETL processes, and big data frameworks.
• Familiarity with unstructured data types and NLP/ML model serving is a plus.
• Knowledge of SQL and NoSQL databases and vector/search technologies.

Responsibilities

• Design high-throughput data pipelines and low-latency data platforms.
• Develop backend microservices for metadata, search, and data retrieval.
• Integrate AI/ML components and real-time model inference into streams.
• Optimize data storage and processing with vector databases and distributed FS.
• Architect APIs and abstractions for diverse data consumer use cases.

Skills

Python/Java/C++

Data Platform Development

Data Engineering

Unstructured Data Handling

Search/NoSQL Databases

Cloud Native (AWS/GCP)

Tools

Kafka

Airflow

Parquet

Arrow

Iceberg

KDB

Elasticsearch

Redis

Solr

MongoDB

AWS

GCP

Job description

Job Summary

Senior Full Stack Data Platform Engineer

Role Overview

We are seeking a Senior Full Stack Software Engineer with deep expertise in building high-throughput data platforms.

In this role, you will architect scalable data platforms using Python, Java, C++, build robust APIs, and enable processing of data using genAI techniques. You will build and optimize a config-driven, plugin-enabled data platform that will allow the construction of DAGs for data processing. You will then apply the platform to build reusable components and pipelines that will ingest gigabytes of unstructured text, audio, and video. You will enable a variety of rich data consumption use-cases by building the right abstractions and APIs for data consumers.

You will be the bridge between complex ML research and real-time trading decisions, working in a poly-language environment (Python, Java, C++) where performance is paramount.

Key Responsibilities

• High-Performance Data Pipelines: Architect low-latency, high-throughput platform that enables rapid development of pipelines to ingest and normalize unstructured data (PDFs, news feeds, audio streams).
• AI ML Integration: Build the infrastructure that wraps and serves NLP and ML models. You will ensure that model inference happens in real-time within the data stream.
• Backend Microservices: Develop robust backend services to handle metadata management, search, and retrieval of processed alternative data.
• System Optimization: Tune the platform for speed. In financial markets, milliseconds matter; you will optimize database queries, serialization, and network calls to ensure data reaches the PMs instantly.
• Data Strategy: Implement storage strategies for unstructured data, utilizing Vector Databases for semantic search and Distributed File Systems for raw storage.

Required Qualifications

• Data Platform Experience: Minimum 5+ years of software engineering experience, preferably building data platforms.
• Core Languages: Strong proficiency in both Python and Java/C++ is required. You should be comfortable switching between these languages for different use cases (e.g., Python for data processing, Java or C++ for high-concurrency, scalable services).
• Data Engineering: Proven experience building data pipelines, ETL processes, or working with big data frameworks (e.g., Kafka, Airflow, Apache Parquet, Arrow, Iceberg, KDB etc).
• Unstructured Data Expertise: Proven experience working with unstructured data types (Text, Audio, Documents). Familiarity with techniques such as OCR, transcription normalization, text extraction.
• Database Knowledge: Proficiency in SQL and significant experience with search/NoSQL engines (Elasticsearch, Redis, Solr, MongoDB or equivalent).
• Cloud Native: Experience building serverless data lakes or processing pipelines on AWS/GCP, etc

Preferred Qualifications

• AI/NLP Exposure: Experience working with Large Language Models (LLMs), Vector Databases (Pinecone, Milvus, Weaviate), or NLP libraries (Hugging Face, spaCy) or similar
• Frontend Competence: Solid experience with modern frontend frameworks (React, Vue, or Angular) and data visualization libraries (e.g., D3.js, Highcharts, or AG Grid).
• Financial Knowledge: Understanding of financial instruments (Equities, Fixed Income) or the investment lifecycle.
• Document Processing: Familiarity with parsing complex document structures (Earnings calls transcripts, 10-K/10-Q filings, Broker Research, Sector and Industry Reports, Central Bank documents, news wires, social media, etc).