Senior Data Engineer

CivicDataLab; New Delhi; Hybrid; INR 2,500,000 - 3,200,000; Full time

Benefits offered by this job

Wellness Allowance

Professional growth

Job summary

CivicDataLab, based in Delhi, seeks a Senior Data Engineer to build scalable data infrastructure and automated ETL pipelines for open data initiatives. You will work with data strategists, researchers, and stakeholders to deliver data-ready datasets and dashboards.

The role emphasizes collaboration, governance, and open-source tooling within a hybrid work model, with travel to project sites as needed.

Qualifications

• Bachelors or higher in CS/Data Science or related field preferred.
• 5+ years of experience as a Data Engineer.
• Proficiency in Python, SQL, and Excel.
• Experience building dashboards (Superset/Tableau/Python visuals).
• Public-sector experience with government or allied institutions is a plus.

Responsibilities

• Develop scalable data infrastructure and data pipelines across platforms.
• Create and oversee data APIs for diverse public data sources.
• Maintain and migrate data platforms like Open Budgets India and Justice Hub.
• Document code, processes, and workflows clearly and thoroughly.
• Lead strategic planning and scoping for data eng projects; align with roadmaps.
• Mentor data engineers; conduct code reviews and capacity-building sessions.
• Perform licensing/ownership checks, PII checks, and metadata quality checks.

Skills

Python

SQL

Excel

Public-sector experience

Agile & Scrum

FOSS contributions

Data modeling

API development

Data pipelines

Mentoring

Education

B.Sc./M.Sc. in Computer Science/Data Science

Tools

Prefect

Airflow

Superset

Tableau

Job description

Job Description - Senior Data Engineer

What We're Looking For:

CivicDataLab is seeking a Senior Data Engineer to support our current and upcoming interventions through open data platforms and open-source tools, focusing on building robust data pipelines for automated data mining, cleaning, standardisation and transformation for AI readiness. We are looking for people who are strongly aligned with our values and have an innate sense of problem-solving, automating processes and adapting well to dynamic environments. They will work alongside data strategists, public policy researchers and other stakeholders to develop automated ETL (Extract, Transform and Load) data pipelines and know how to model, store and manage bulk datasets. This will help us co‑create comprehensive data analytics tools and dashboards for our diverse stakeholders.

At CivicDataLab, we believe in using data, tech, design, and social science to strengthen civic engagement and drive evidence-based decision‑making. Our projects centre on building data strategy, platforms, and applications to foster data‑driven governance.

Key Responsibilities:

• Take ownership over developing scalable data infrastructure for our existing platforms and products from data orchestration pipelines using Prefect and Airflow, to Data Sharing Protocols for our open data collaboratives.
• Create and oversee data APIs responsible for collecting, managing, and analysing data from diverse public data sources.
• Maintain and monitor our existing open data platforms like Open Budgets India, Justice Hub, Open Contracting India, and oversee dataset migration into new platforms.
• Thoroughly document code, processes, and all activities performed by the data team, ensuring clarity and comprehensiveness. This includes documenting algorithms, methodologies, data transformations, and the overall workflow.
• Lead strategic planning and scoping for data engineering projects, aligning them with product roadmaps and stakeholder needs.
• Mentor and provide technical guidance to a team of data engineers, conducting code reviews and capacity‑building sessions.
• Quality & compliance: Run structured checklists (licensing/ownership validation, PII checks, metadata completeness) and coordinate fixes with publishers.

Requirements

Recommended Skills & Requirements:

• Any graduate, however, candidates with a B.Sc./M.Sc. (or higher) in Computer Science, Data Science, or a related field will be preferred.
• 5+ Years of experience as a Data engineer.
• Proficiency in Python, SQL, and Excel.
• Experience building dashboards (Superset, Tableau, or Python frameworks).
• Excellent written/verbal communication and documentation skills.
• Public-sector experience: Demonstrated work with government departments/PMUs or allied institutions; comfortable with consultations and public‑sector processes.
• Onboarding literacy: Familiarity with dataset discovery, metadata, licensing/ownership, and basic PII/privacy checks; rigorous documentation and follow‑through.
• Operations & collaboration: Excellent coordination and communication skills; readiness for periodic intra-state travel.
• Extensive working knowledge of API or Stream‑based data extraction processes.
• Ability to gauge requirements for the project, make decisions and coordinate with other team members.
• Prior experience in actively working and contributing to FOSS (Free and Open Source Software) communities.
• Familiarity with working with Agile methodologies and Scrum processes.
• Experience with scalable infrastructure practices such as microservice architecture, infrastructure as code, distributed systems, scaling methods, load balancing and more.

How we work

CivicDataLab is based out of Delhi and has the presence of project teams located in Assam and Himachal Pradesh. We follow a hybrid model where our bandhus work out of the office for a minimum of 12 days per month (i.e.) 3 days a week. We use open‑source tools and agile methodologies in organising our work.

Perks of Working with Us

Wellness Allowance

At CivicDataLab, we always emphasise the wellness of our bandhus. This includes any Expenditure done for the purpose of Wellness Setup, except any financial instrument, any expense that can be claimed as a deductible expense under Income Tax rules, any goods and services that attract a combined tax, cess or duty of more than 28%. If you're interested in taking classes that enhance your overall physical or mental well-being, you have an INR 60,000 annual stipend to do so. For some people, that might mean a monthly massage. Some take photography lessons or learn a musical instrument or buy a gym membership. It's up to you; the point is to learn something that you feel enriches you as a person.

Professional growth and development Allowance

At CivicDataLab, we encourage everyone to take up things that help one grow professionally, and you get an annual kitty of INR 60,000 to do so. This includes attending or speaking at conferences and workshops, taking courses, acquiring hardware or software licenses or even joining summer schools. We feel that learning a skill should never be a hurdle to solve important problems for the community.