Lead Data and AI Architect

Pratham Software Inc.; Jaipur; On-site; INR 350,000 - 550,000; Full time

Job summary

Pratham Software Inc. is seeking a Lead Data Architect to design, build, and scale data pipelines and entity resolution systems. You will lead architectural decisions, mentor engineers, and ensure pipelines are reliable and production-grade.

The role combines data engineering expertise with hands-on AI-assisted tooling, entity matching, and data integration from diverse sources. Collaboration with cross-functional teams is essential to translate business requirements into scalable data

Qualifications

• Strong hands-on experience building production-grade data pipelines at scale.
• Practical experience with Dagster (or similar orchestration tools like Airflow/Prefect).
• Experience with Splink or similar probabilistic/deterministic record linkage tools for entity matching.
• Strong proficiency in Python, including experience writing and maintaining web scrapers.
• Proven experience integrating and maintaining data pipelines from multiple sources.
• Experience applying AI/ML tools within data engineering workflows (LLM-assisted data cleaning, extraction, or matching).
• Hands-on experience with PostgreSQL, Google Cloud Spanner, and other relational/distributed databases.

Responsibilities

• Architect, build, and maintain robust, scalable data pipelines that ingest, transform, and serve data from multiple sources.
• Own the end-to-end orchestration of data workflows using Dagster, ensuring observability, reliability, and maintainability of pipelines.
• Design and implement entity resolution workflows using Splink to produce clean, deduplicated, golden records.
• Build and maintain web scrapers to source data from external providers, handling rate limits and data quality issues.
• Integrate data from multiple, often inconsistent, sources into unified datasets.
• Design and maintain data models and schemas across transactional and analytical systems for scalability and performance.
• Leverage AI/LLM-based tools to enhance data pipeline capabilities (e.g., intelligent extraction, automated quality checks).
• Define and enforce best practices around pipeline design, testing, monitoring, and documentation.
• Collaborate with data engineers, product managers, and stakeholders to translate requirements into scalable architecture.
• Provide technical leadership and mentorship to the data engineering team.

Job description

We are looking for a Lead Data Architect to design, build, and scale our data pipelines and entity resolution systems. This role combines deep technical expertise in data engineering with hands‑on experience in AI-assisted tooling, entity matching, and data integration from diverse sources. You will lead architectural decisions for our data platform, mentor engineers, and ensure our pipelines are reliable, scalable, and production‑grade.

Responsibilities

• Architect, build, and maintain robust, scalable data pipelines that ingest, transform, and serve data from multiple internal and external sources.
• Own the end-to-end orchestration of data workflows using tools like Dagster, ensuring observability, reliability, and maintainability of pipelines.
• Design and implement entity resolution workflows - including matching, merging, and survivorship logic - using tools such as Splink, to produce clean, deduplicated, golden records.
• Build and maintain web scrapers to source data from external providers, ensuring resilience to source changes, rate limits, and data quality issues.
• Integrate and reconcile data coming from multiple, often inconsistent, sources into unified, trustworthy datasets.
• Design and maintain data models and schemas across transactional and analytical systems, ensuring consistency, scalability, and performance.
• Leverage AI/LLM-based tools and techniques to enhance data pipeline capabilities - eg, intelligent data extraction, automated data quality checks, or AI-assisted entity matching.
• Define and enforce best practices around pipeline design, testing, monitoring, and documentation.
• Collaborate closely with data engineers, product managers, and other stakeholders to translate business requirements into scalable data architecture.
• Provide technical leadership and mentorship to the data engineering team.

Requirements

• Strong hands‑on experience building and maintaining production‑grade data pipelines at scale.
• Practical experience with Dagster (or similar orchestration tools like Airflow/Prefect) for pipeline orchestration.
• Experience with Splink or similar probabilistic/deterministic record linkage tools for entity matching, merging, and survivorship.
• Strong proficiency in Python, including experience writing and maintaining web scrapers.
• Proven experience integrating and maintaining data pipelines that pull from multiple, heterogeneous data sources.
• Experience applying AI/ML tools within data engineering workflows (eg, LLM‑assisted data cleaning, extraction, or matching).
• Hands‑on experience with relational and distributed databases such as PostgreSQL and Google Cloud Spanner.
• Strong understanding of data modeling principles (normalization, dimensional modeling, schema design) across OLTP and OLAP systems.
• Experience with cloud data warehousing platforms such as BigQuery, Redshift, and cloud platforms (GCP/AWS/Azure).
• Strong communication skills and experience working cross‑functionally with engineering and product teams.
• Experience with distributed data processing frameworks (eg, Spark, Dask).
• Familiarity with data governance, lineage, and cataloging tools.
• Prior experience in a lead or architect‑level role guiding a data engineering team.

What Were Looking For

A technically strong, hands‑on leader who can balance architectural thinking with the practical grit of debugging a flaky scraper or tuning a matching algorithm - someone whos comfortable owning both the big picture and the messy details of real‑world data.

Disclaimer: This job posting has been aggregated from external source. Role details, content, and availability are subject to change. Applicants are advised to confirm the latest information directly on the company website before applying.