Software Engineer, Data Infrastructure and Acquisition

Analogy Group; India; On-site; INR 3,500,000 - 6,000,000; Full time

Job summary

Analogy Group is hiring a Software Engineer focused on Data Infrastructure and Acquisition to design the infrastructure and build scrapers for public data. This fully remote role is open to engineers based in India, with a strong track record of technical excellence welcomed from top programs, tech companies, startups, or independent work.

You will design, build, and maintain data pipelines using Prefect, AWS, and Django; create durable scraping abstractions; develop AI-powered agents and

Qualifications

• Bachelor's or Master's degree in Computer Science or Engineering.
• 5+ years of professional software engineering experience.
• Strong Python skills and working knowledge of JavaScript/TypeScript.
• Experience designing and operating complex data pipelines.
• Experience with workflow orchestration platforms (Temporal, Prefect, Dagster, Airflow).

Responsibilities

• Design, build, and maintain data acquisition infrastructure using Prefect, AWS, and Django.
• Develop durable abstractions for scraping patterns and scalable architectures.
• Build AI-powered infrastructure for agentic, self-healing scrapers.
• Develop agents capable of creating and validating new scrapers from templates.
• Create resilient scrapers for messy public data sources (PDFs, forms, portals).
• Improve observability, retries, validation, and failure recovery across pipelines.
• Open-source selected parts of our work and write about learnings.

Skills

Python

JavaScript/TypeScript

Data pipelines

Web scraping

AI tooling

Observability

Testing

Software design

Education

Bachelor's or Master's in CS/Engineering

Tools

Prefect

AWS

Django

Temporal

Dagster

Airflow

Codex

Claude Code

Job description

Position: Software Engineer, Data Infrastructure and Acquisition

Analogy Group builds cutting-edge AI political intelligence tools. We work with advocacy organizations, researchers, and companies working on the most important problems in the world—from climate to democracy to AI policy. We use LLMs to make sense of messy data, enabling these organizations to move faster with more informed strategy.

We are hiring a Software Engineer focused on Data Infrastructure and Acquisition to help us design the infrastructure and build scrapers for public data. This role is a mix of backend engineering, agentic AI engineering, and web scraping. We want to make government data accessible and useful to help fight corruption, protect democracy, and make citizens able to understand their government, and your work will directly contribute to this mission.

This is a fully remote role for engineers based in India. We offer highly competitive, top-of-market compensation and are looking for candidates with a strong record of technical excellence, whether developed at a top engineering program, a high-performing technology company, an ambitious startup, or through exceptional independent work.

What You’ll Do

• Design, build, and maintain our data acquisition infrastructure using technologies such as Prefect, AWS, and Django.
• Develop clear, durable abstractions for scraping patterns and infrastructure, keeping the system manageable as we scale to thousands of data sources.
• Build AI-powered infrastructure for agentic, self-healing scrapers.
• Develop agents capable of creating new scrapers from established templates and validating their output.
• Build and test resilient scrapers for messy, difficult public data sources, including PDFs, videos, forms, and legacy government portals.
• Improve observability, retries, data validation, and failure recovery across our pipelines.
• Open-source selected parts of our work and write about what we learn.

Minimum Requirements

• Bachelor's or Master's degree in Computer Science, Engineering, or similar.
• 5+ years of professional software engineering experience, with strong Python skills and working knowledge of JavaScript or TypeScript.
• Experience designing and operating complex data pipelines.
• Experience with workflow orchestration platforms such as Temporal, Prefect, Dagster, or Airflow.
• Practical web-scraping experience, including headless browsers, retries, rate limiting, proxy rotation, and changing or unreliable source websites.
• Experience using coding agents such as Codex or Claude Code, including an understanding of what makes agent workflows reliable.
• Experience developing AI agents, agent tooling, evaluation systems, skills, or execution harnesses.
• Strong judgment about abstractions, testing, observability, and long-term system maintainability.

Nice to Have

• Experience extracting structured information from PDFs, scanned documents, audio, or video.
• Experience operating distributed systems or data infrastructure on AWS.
• Experience with Django and Prefect.
• Contributions to open-source software or published technical writing.
• An interest in government transparency, public-interest technology, politics, or civic data.