Lead Data / AI Architect

Resources Valley; Jaipur; On-site; INR 1,800,000 - 2,800,000; Full time

Job summary

Resources Valley in Jaipur seeks a Lead Data Architect to design, build, and scale data pipelines and entity resolution systems, blending data engineering with AI-assisted tooling. You will lead architectural decisions for our data platform, mentor engineers, ensure reliability, scalability, and production-grade quality across diverse data sources.

The role emphasizes collaboration with product and engineering teams, implementing best practices, and leveraging cloud data warehousing and modern

Qualifications

• Experience building production-grade data pipelines at scale.
• Hands-on with Dagster or similar orchestration tools.
• Experience with Splink for entity matching and survivorship.
• Strong Python development and data modeling skills.

Responsibilities

• Architect, build, and maintain robust, scalable data pipelines that ingest, transform, and serve data from multiple internal and external sources.
• Own the end-to-end orchestration of data workflows using Dagster, ensuring observability, reliability, and maintainability of pipelines.
• Design and implement entity resolution workflows, including matching, merging, and survivorship logic, using Splink.
• Build and maintain web scrapers to source data from external providers, ensuring resilience to source changes, rate limits, and data quality issues.
• Integrate and reconcile data from multiple sources into unified, trustworthy datasets.
• Design and maintain data models and schemas across transactional and analytical systems, ensuring consistency, scalability, and performance.
• Leverage AI/LLM-based tools to enhance data pipeline capabilities, e.g., intelligent data extraction, automated data quality checks, or AI-assisted entity matching.
• Define and enforce best practices around pipeline design, testing, monitoring, and documentation.
• Collaborate closely with data engineers, product managers, and other stakeholders to translate business requirements into scalable data architecture.
• Provide technical leadership and mentorship to the data engineering team.

Skills

Python

Dagster

Airflow

Splink

Data pipelines

Data modeling

Cloud data warehousing

PostgreSQL

BigQuery

Redshift

Spark

Dask

Leadership

Cross-functional collaboration

Tools

Dagster

Airflow

Prefect

Splink

PostgreSQL

BigQuery

Redshift

Spark

Dask

Job description

We are looking for a Lead Data Architect to design, build, and scale our data pipelines and entity resolution systems. This role combines deep technical expertise in data engineering with hands‑on experience in AI-assisted tooling, entity matching, and data integration from diverse sources. You will lead architectural decisions for our data platform, mentor engineers, and ensure our pipelines are reliable, scalable, and production-grade.

Key Responsibilities

• Architect, build, and maintain robust, scalable data pipelines that ingest, transform, and serve data from multiple internal and external sources.
• Own the end‑to‑end orchestration of data workflows using tools like Dagster, ensuring observability, reliability, and maintainability of pipelines.
• Design and implement entity resolution workflows, including matching, merging, and survivorship logic, using tools such as Splink, to produce clean, deduplicated, golden records.
• Build and maintain web scrapers to source data from external providers, ensuring resilience to source changes, rate limits, and data quality issues.
• Integrate and reconcile data coming from multiple, often inconsistent, sources into unified, trustworthy datasets.
• Design and maintain data models and schemas across transactional and analytical systems, ensuring consistency, scalability, and performance.
• Leverage AI/LLM‑based tools and techniques to enhance data pipeline capabilities, e.g., intelligent data extraction, automated data quality checks, or AI‑assisted entity matching.
• Define and enforce best practices around pipeline design, testing, monitoring, and documentation.
• Collaborate closely with data engineers, product managers, and other stakeholders to translate business requirements into scalable data architecture.
• Provide technical leadership and mentorship to the data engineering team.

Required Skills & Experience

• Strong hands‑on experience building and maintaining production‑grade data pipelines at scale.
• Practical experience with Dagster (or similar orchestration tools like Airflow/Prefect) for pipeline orchestration.
• Experience with Splink or similar probabilistic/deterministic record linkage tools for entity matching, merging, and survivorship.
• Strong proficiency in Python, including experience writing and maintaining web scrapers.
• Proven experience integrating and maintaining data pipelines that pull from multiple, heterogeneous data sources.
• Experience applying AI/ML tools within data engineering workflows (e.g., LLM‑assisted data cleaning, extraction, or matching).
• Hands‑on experience with relational and distributed databases such as PostgreSQL and Google Cloud Spanner.
• Strong understanding of data modeling principles (normalization, dimensional modeling, schema design) across OLTP and OLAP systems.
• Experience with cloud data warehousing platforms such as BigQuery, Redshift, and cloud platforms (GCP/AWS/Azure).
• Strong communication skills and experience working cross‑functionally with engineering and product teams.
• Experience with distributed data processing frameworks (e.g., Spark, Dask).
• Familiarity with data governance, lineage, and cataloging tools.
• Prior experience in a lead or architect‑level role guiding a data engineering team.