Data Engineering Lead

National e-Governance Division

New Delhi

On-site

INR 3,000,000 - 7,000,000

Full time

14 days+

Get a reply from this employer

Your resume and cover letter, tailored to exactly what they’re hiring for.

Get past ATS filters

Job summary

National e-Governance Division seeks a Data Engineering Lead to build and run data foundations for AI services, including ingestion from government systems, document pipelines, data platform management, and versioned training datasets. The role involves hands-on work across on-premise and government cloud infrastructure, with emphasis on governance and privacy.

The candidate will lead data standards, collaborate with vendor teams, and ensure reliable, high-volume pipelines for analytics and AI

Qualifications

  • 8+ years in data engineering or data platform work (10+ years preferred).
  • Minimum 4 years designing and running large-scale data pipelines for analytics or AI systems.
  • Ownership of at least one production data platform with volumes, reliability and data-quality measures.
  • Experience with structured, semi-structured and unstructured data, including documents/images.

Responsibilities

  • Build batch, streaming and change-data-capture ingestion from databases, APIs and files; connect to national platforms.
  • Develop high-volume intake for scanned documents/images with parsing and OCR pipelines with NLP & Document AI team; capture metadata.
  • Run lakehouse/warehouse with orchestration, transformation, quality checks, lineage and data catalogue on IndiaAI and on-premise infra.
  • Maintain registry of training data and golden evaluation datasets with versioning and access control; run annotation pipelines for Indian-language data.
  • Build embedding pipelines and manage vector indexes; synchronize with AIKosh catalogue and prepare datasets for publication.
  • Apply classification, masking, anonymisation and retention of personal data under DPDPA 2023; maintain access logs.
  • Build pipelines, set data standards for vendor teams and review pipelines before acceptance.

Skills

Data pipelines
On-prem & cloud infra
Data governance & privacy
Vector indexes / embeddings
NLP / Document AI

Education

B.Tech./B.E., M.Tech./M.S. or M.Sc. in Computer Science, Data Engineering, Information Technology or related
Certification in data platforms or Kubernetes
Non-traditional backgrounds with verifiable work

Tools

Kubernetes
AIKosh

Job description

Data Engineering Lead

Role Summary

The Data Engineering Lead will build and run the data foundations for AI services. This covers ingestion from departmental systems and national platforms, document pipelines at volume, the data platform, versioned training and evaluation datasets, retrieval (RAG) infrastructure, and synchronisation with AIKosh. The role is hands‑on and sets data standards for in‑house and vendor teams. It requires strong engineering on on‑premise and government cloud infrastructure, not dependence on any single commercial data warehouse.


Educational Qualification

  • B.Tech./B.E., M.Tech./M.S. or M.Sc. in Computer Science, Data Engineering, Information Technology or a related discipline (Must have)
  • Professional certification in data platforms or Kubernetes is desirable but not mandatory; demonstrated work, as listed under Minimum Evidence at Screening, shall carry greater weight
  • Candidates from non‑traditional backgrounds with strong verifiable work will be considered in lieu of formal qualification

Experience
  • 8+ years in data engineering or data platform work (10+ years preferred); minimum 4 years designing and running large‑scale data pipelines for analytics or AI systems
  • Demonstrable ownership of at least one production data platform, with documented volumes, reliability and data‑quality measures
  • Experience with structured, semi‑structured and unstructured data, including documents and images at volume
  • Experience building training and evaluation datasets with ML teams; experience with embedding and vector‑index pipelines for retrieval is a strong plus
  • Experience on both on‑premise and cloud infrastructure
  • Experience handling personal data under data‑protection law is a strong plus

Key Responsibilities

Data Acquisition & Integration
  • Build batch, streaming and change‑data‑capture ingestion from departmental databases, APIs and files, and connectors to national platforms
  • Define the technical terms of data‑sharing agreements with departments
Document & Unstructured Data Pipelines
  • Build high‑volume intake for scanned documents and images, with parsing and OCR pipelines run with the NLP & Document AI team, metadata capture and storage
Data Platform
  • Run the lakehouse or warehouse, orchestration, transformation, quality checks, lineage and data catalogue on IndiaAI and on‑premise infrastructure
Datasets for AI
  • Maintain a registry of training data and golden evaluation datasets, with versioning, dataset documentation and access control
  • Run annotation pipelines, including for Indian‑language data, with the QA team
Retrieval Infrastructure & AIKosh
  • Build embedding pipelines and manage the lifecycle of vector indexes, including separate indexes per department
  • Synchronise with the AIKosh catalogue, and prepare datasets for publication on AIKosh
Data Governance & Privacy
  • Apply classification, masking, anonymisation and retention of personal data under DPDPA 2023, and maintain access logs
Leadership & Standards
  • Build pipelines directly, set data standards for vendor teams, and review their pipelines before acceptance
Minimum Evidence at Screening (one required)
  • A production data platform or pipelines the candidate built, with documented volumes and reliability, verifiable through reference or artefact
  • A data‑quality, lineage or dataset‑versioning approach the candidate put in place, with its results
  • An open‑source contribution to a data engineering tool, with verifiable commit history

Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Senior Data Platform Engineer
Senior Data Platform Engineer

Relevance Lab • Bengaluru

On-site
INR 400,000 - 700,000
Data Engineer, Data Platforms
Data Engineer, Data Platforms

S&P Global • Hyderabad

On-site
INR 1,500,000 - 2,100,000
Senior Data Engineer
Senior Data Engineer

Backbase • Hyderabad

On-site
INR 4,000,000 - 7,000,000
Data Engineer - Senior 1
Data Engineer - Senior 1

Fa Espx Saasfaprod1 • Pune District

On-site
INR 1,200,000 - 1,800,000
Data Engineer 11
Data Engineer 11

TigerHire • Pune District

On-site
INR 1,500,000 - 2,300,000
Lead Data Engineer (Lead Software Engineer)
Lead Data Engineer (Lead Software Engineer)

DigitalXNode • Ahmedabad District

On-site
INR 1,800,000 - 2,600,000
Data Engineer
Data Engineer

Thalus Technologies Pvt Ltd • Margao

On-site
INR 1,200,000 - 2,500,000
AI CoE Senior Data Engineer & Architect
AI CoE Senior Data Engineer & Architect

Qadinc • Pune

On-site
INR 2,600,000 - 4,200,000
Technical Lead - Data Engineer (Data&AI)
Technical Lead - Data Engineer (Data&AI)

Srijan: Now Material • Gurugram District

On-site
INR 4,000,000 - 7,500,000
Data Engineer
Data Engineer

Valuenode Private Limited • India

Remote
INR 1,200,000 - 1,800,000