Senior Data Engineer

Billennium

India

Remote

INR 4,000,000 - 7,000,000

Full time

12 days ago

A complete application in a minute

Tailored resume and cover letter, ready to send.

Get past ATS filters

Job summary

Billennium is seeking aSenior AI Data Engineer / Data Scientist to turn messy enterprise data into AI-ready knowledge assets. You will own data quality end-to-end, from discovery and cleanup to enrichment and governance, working with AI engineers and business stakeholders to determine what data is worth using and how to structure it for retrieval and reasoning.

The role uses a standardized stack including Postgres + pgvector, Redis, LLM gateway patterns, Langfuse observability, and privacy

Qualifications

  • 5+ years in data engineering / analytics with ownership of production pipelines.
  • Strong Python for data processing.
  • Experience with unstructured enterprise data (documents, PDFs, Office files, wikis).
  • Experience building data pipelines for retrieval/NLP/RAG use cases.
  • Solid understanding of data quality engineering: validation, monitoring, lineage, refresh cycles.
  • Strong stakeholder collaboration with business to define what data matters.

Responsibilities

  • Lead data discovery, cleanup, and triage for AI use cases.
  • Own end-to-end data quality outcomes and governance.
  • Build ingestion pipelines for SharePoint and document repositories; metadata capture.
  • Design retrieval- and RAG-optimized data structures and chunking strategies.
  • Collaborate with AI Engineers to evaluate retrieval quality and trust metrics.
  • Ensure privacy and governance for enterprise AI data assets.

Skills

Python
Data pipelines
NLP/RAG
Data quality
Stakeholder mgmt

Tools

Postgres + pgvector

Job description

We are looking for a Senior AI Data Engineer / Data Scientist who can turn messy enterprise data into AI-ready, high-quality knowledge assets. You will lead the cleanup, preparation, and enrichment of unstructured content (SharePoint/document repositories) and structured/semi-structured data (data lakes, databases) so our agents, copilots, and RAG systems are accurate, trustworthy, and scalable.


This is a senior, hands-on role. You will own data quality outcomes end-to-end: discovery - cleanup - enrichment - ingestion - refresh cycles - governance. We value AI-native generalists who can remove bottlenecks by working directly with AI Engineers, Architects, and business stakeholders to decide what data is worth using and how to structure it for retrieval and reasoning.


Our standardized stack includes (and this role actively uses it): ingestion/ETL foundations, Postgres + pgvector as default RAG store, Redis caching, LLM gateway patterns, Langfuse observability, DeepEval/RAGAS evaluation, and Presidio for PII detection/masking when required.


KEY RESPONSIBILITIES

  • Enterprise data discovery & cleanup
  • Lead data triage for AI use cases: identify authoritative sources, duplicates, outdated content, and low-quality documents.
  • Clean, normalize, deduplicate, and standardize enterprise content at scale (documents, PDFs, Word/Excel, wiki pages, etc.).
  • Define what data should be excluded from AI systems (stale, contradictory, low-trust, or sensitive content).
  • Unstructured ingestion (SharePoint + document repositories)
  • Build robust ingestion pipelines for SharePoint and file repositories: parsing, text extraction, structure recovery, and metadata capture.
  • Implement document normalization strategies (naming, taxonomy, metadata standards, canonical IDs).
  • RAG-ready data engineering (context engineering, not just ETL)
  • Design chunking strategies, metadata enrichment, and document structuring optimized for retrieval performance and cost.
  • Improve retrieval quality through practical techniques such as filtered retrieval and post-retrieval optimization where appropriate (e.g., reranking), collaborating with AI Engineers on the retrieval interface.
  • Prepare and maintain AI-ready knowledge sets that can be embedded and served via Postgres + pgvector (default).
  • Data quality, evaluation, and feedback loops (non-negotiable)
  • Define and implement data quality gates (freshness, completeness, relevance, dedupe rate, metadata coverage).
  • Partner with AI Engineers to evaluate retrieval and RAG performance using frameworks like RAGAS (answer correctness, context recall/precision) and to monitor trust metrics over time.
  • Establish human feedback loops where needed (review queues, sampling, targeted audits) to continuously improve data usefulness and user trust.
  • Governance, privacy, and auditability
  • Ensure traceability: lineage from source processed artifact retrieval corpus production usage.
  • Apply privacy and enterprise constraints; where required, implement PII detection/masking using Presidio patterns.
  • Reuse
  • Package reusable data cleanup + RAG readiness recipes: ingestion templates, metadata schemas, chunking playbooks, dedupe strategies.
  • Build a repeatable data foundation that accelerates future use cases (not a one-off cleanup project).

REQUIRED EXPERIENCE (MUST-HAVE)

  • 5+ years in data engineering / applied data science / analytics engineering with ownership of production pipelines.
  • Strong Python for data processing (parsing, cleaning, normalization, pipelines).
  • Proven experience working with unstructured enterprise data (documents, PDFs, Office files, wikis, knowledge bases).
  • Proven experience building data pipelines for retrieval/NLP/RAG-like use cases (embedding prep, metadata, corpus management).
  • Solid understanding of data quality engineering: validation, monitoring, lineage, refresh cycles.
  • Strong stakeholder skill: can work with business to define what data matters and what “good” looks like.

NICE-TO-HAVE

  • Experience with Postgres + pgvector (or similar vector stores), retrieval optimization, and hybrid search concepts.
  • Familiarity with observability practices for AI pipelines and the use of RAG evaluation metrics (RAGAS-style).
  • Experience with governance tooling and privacy controls for enterprise AI (e.g., PII workflows).
Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Senior Data Scientist – AI & GenAI
Senior Data Scientist – AI & GenAI

D4 Insight • Hyderabad

On-site
INR 3,000,000 - 6,000,000
Data Engineer
Data Engineer

Techdome • Hyderabad

On-site
INR 1,500,000 - 2,200,000
Data & Integration Analyst
Data & Integration Analyst

Scalis LLC • Pune District

On-site
INR 1,200,000 - 1,800,000
Senior AI Engineer
Senior AI Engineer

TheMathCompany Inc. • Bengaluru

On-site
INR 2,400,000 - 4,200,000
Data Engineer - Senior 1
Data Engineer - Senior 1

Fa Espx Saasfaprod1 • Pune District

On-site
INR 1,200,000 - 1,800,000
Senior Intergration AI Engineer
Senior Intergration AI Engineer

Ensylon • Pune

On-site
INR 3,500,000 - 5,500,000
Senior Expert, IT Development
Senior Expert, IT Development

Randstad Enterprise • Pune District

On-site
INR 4,000,000 - 7,000,000
Senior Data Architect
Senior Data Architect

Zeus Ai Llp • Hyderabad

Hybrid
INR 4,000,000 - 7,000,000
Data Engineer Staff
Data Engineer Staff

Hartford • Hyderabad

Hybrid
INR 1,200,000 - 2,000,000
senior expert, it development
senior expert, it development

Randstad NV • Pune District

On-site
INR 3,000,000 - 5,400,000