Principal Data Consultant

Autonomai Recruitment; Maharashtra; On-site; INR 4,000,000 - 6,500,000; Full time

Job summary

Autonomai Recruitment is seeking a Principal Data Engineer to design and build a next-generation data platform for a global trading firm in Maharashtra, India. You will ingest, normalize and deliver large proprietary datasets with low latency, collaborating with traders, quants and business stakeholders.

You will lead data engineering efforts, build automation, and apply AI/LLM techniques to extract structured information from diverse sources.

Qualifications

• Significant experience as a Principal/Lead Data Engineer in trading/fintech.
• Expert-level Python with production-grade data systems.
• Strong SQL and data-modelling across relational/NoSQL stores.
• Experience building data platforms from scratch for high-volume data.

Responsibilities

• Define architecture for a firm-wide data platform, including ingestion, storage and delivery.
• Build high-volume batch and near-real-time data pipelines for multiple datasets.
• Design data models for market, reference, alternative and proprietary data.
• Ingest and enrich unstructured sources like documents and filings.
• Collaborate with traders and quants to translate needs into data products.
• Develop production-grade Python services and pipeline components.
• Apply LLMs/AI techniques to extract structured information from sources.
• Implement data quality, validation, lineage and reconciliation frameworks.
• Mentor junior data engineers and collaborate across teams.
• Stay hands-on: write code, debug pipelines, review designs.

Skills

Python

SQL

Data modelling

NoSQL

Tools

ETL pipelines

DAG orchestration

LLMs/AI agents

Job description

Team: Data Platform / Trading Technology

We are seeking a Principal Data Engineer to design and build a next-generation proprietary data platform from the ground up at a global leading trading firm. You will create the systems, pipelines and automation needed to ingest, normalise, validate and deliver very large volumes of proprietary and alternative data across HFT, ETFs, options, indices, ecommerce, contracts and M&A.

This is a high-impact, hands-on leadership role. You will work directly with traders, quantitative researchers, engineers and business stakeholders to turn fragmented, complex and often unstructured data into reliable, low-latency, research-ready and trading-ready datasets.

You will combine strong data engineering with automation, Python development and practical use of LLMs and AI agents to accelerate ingestion, extraction, enrichment, classification and data-quality workflows.

What You’ll Do

• Define and deliver the technical architecture for a firm-wide proprietary data platform, covering ingestion, storage, transformation, cataloguing, lineage, access and delivery.
• Build high-volume, high-reliability batch and near-real-time data pipelines for market, reference, alternative and proprietary datasets.
• Design pipelines and data models for HFT-related data, including market data, order data, execution data, ETF, options and index data.
• Build data ingestion and enrichment systems for ecommerce, corporate contracts, M&A, filings, news and other unstructured or semi-structured sources.
• Work closely with traders and quants to understand research, alpha, risk, execution and trading-workflow requirements, then translate them into scalable data products.
• Build automated ingestion, parsing, normalisation, deduplication, validation and reconciliation frameworks.
• Develop production-grade Python services, tooling and pipeline components for data extraction, transformation, orchestration and quality control.
• Apply LLMs, embeddings and AI-agent workflows to extract structured information from documents, contracts, filings, web content and other unstructured sources.
• Build automated data-quality, anomaly-detection and lineage frameworks to ensure datasets
• Mentor data engineers and collaborate with infrastructure, quant, trading and software engineering teams.
• Stay hands-on: write code, debug complex pipeline issues, review designs and drive technical decisions.

What We’re Looking For

• Significant experience as a Principal, Staff, Lead or Senior Data Engineer, ideally within a trading firm, hedge fund, market maker, fintech or data-intensive technology business.
• Expert-level Python, with a track record of building clean, performant, production-grade data engineering systems.
• Deep experience designing and operating large-scale ETL/ELT pipelines across multiple internal and external data sources.
• Strong SQL and data-modelling skills, with experience across relational, columnar and/or NoSQL databases.
• Experience building data platforms or critical data infrastructure from scratch, rather than only maintaining existing systems.
• Strong understanding of data quality, reconciliation, lineage, metadata, schema evolution and automated validation.
• Experience processing structured, semi-structured and unstructured data, including APIs, files, web data, documents and event streams.
• Practical experience using LLMs or AI/ML techniques for extraction, classification, enrichment, search or workflow automation.
• Direct experience with financial datasets, including equities, ETFs, options, futures, indices, market data, reference data or alternative data.
• Exposure to HFT, electronic trading, execution, order-book or market-microstructure data.
• Experience with low-latency or high-throughput data systems.