Data Platform Engineer

Uplers Solutions Private Limited.; Gurugram District; Hybrid; INR 4,000,000 - 6,500,000; Full time

Job summary

1DigitalStack.ai is seeking a Data Platform Engineer in Gurugram for a senior individual contributor role. The platform supports 1P marketplace crawls across 220+ global marketplaces, shifting from crawler-by-crawler work to scalable platform ownership.

You will own orchestration, data quality, observability, and service integration, building a reliable, scalable system with architectural ownership.

Qualifications

• 5–8 years building production Python systems.
• Deep hands‑on Scrapy and Playwright or Selenium at scale.
• Experience with authenticated portals, session/token lifecycles.
• Strong distributed queueing with RabbitMQ or Kafka.
• Workflow orchestration with Airflow/Temporal/Prefect or Kestra.
• Production experience with MongoDB and PostgreSQL; index design.
• Observability with OpenTelemetry, Prometheus, Grafana.
• Strong HTTP/HTML/CSS, Linux, Docker, CI/CD fundamentals.
• Ability to reason about distributed failures across services.

Responsibilities

• Own scheduling and orchestration across marketplaces, accounts, regions.
• Design retry, backoff, dead-letter, and idempotent rerun semantics.
• Build heartbeat, timeout, and automated recovery for long‑running jobs.
• Manage concurrency, rate limiting, session lifecycle, credential rotation.
• Define completion SLAs and design the platform to meet them.
• Design schema contracts, data validation gates, and deduplication.
• Instrument platform with logging, metrics, tracing; ensure traceability.
• Integrate pipeline with auth, relational, and analytical services.

Skills

Python

Asyncio

Scrapy

Playwright/Selenium

Distributed queues

RabbitMQ/Kafka

Workflow orchestration

Airflow/Temporal/Prefect

MongoDB

PostgreSQL

OpenTelemetry/Prometheus/Grafana

HTTP/HTML/CSS

Linux

Docker/CI/CD

Distributed failure analysis

Tools

MongoDB

PostgreSQL

RabbitMQ

Kafka

Airflow

Temporal

Prometheus

Grafana

Docker

CI/CD pipelines

Job description

Job Description:

Data Platform Engineer

Note: This is a requirement for one of Uplers' client - 1digitalstack.ai

Location

Gurugram

Shift

(GMT+05:30) Asia/Kolkata (IST)

Opportunity Type

Hybrid ()

Employment Type

Full-Time — Hybrid

Contract

Full time Permanent Position

Salary

Confidential (based on experience)

Experience

5 to 8 years

5.00 + years

Role Description

We are hiring a senior individual contributor to help scale the platform behind our first‑party (1P) marketplace data operations. We run high‑volume authenticated crawls across 220+ global marketplaces. As that footprint grows, the engineering challenge shifts from writing individual crawlers to building the platform that runs thousands of them predictably. This role sits at that layer. You will work on orchestration, data quality, observability, and service integration — the systems that let a large crawler fleet run at a known standard rather than case by case. It is a hands‑on engineering role with real architectural ownership.

What You Will Own

• Orchestration and job reliability
  • Own the scheduling and orchestration layer that coordinates 1P crawl jobs across marketplaces, accounts, and regions.
  • Design retry, backoff, dead‑letter, and idempotent rerun semantics so repeated execution is always safe.
  • Build heartbeat, timeout, and automated recovery patterns into long‑running distributed jobs.
  • Manage concurrency, rate‑limiting, session lifecycle, and credential rotation for authenticated portals.
  • Define completion SLAs by job type and design the platform to meet them.
• Data integrity
  • Design schema contracts and validation gates that data passes through before publication.
  • Build deduplication into the pipeline using natural keys, content hashing, and idempotency keys.
  • Implement completeness checks across marketplace, account, and date dimensions.
  • Apply anomaly detection to volume, field‑fill rates, and value distributions.
  • Design quarantine and replay paths so questionable batches are held and reprocessed cleanly.
• Observability
  • Instrument the platform with structured logging, metrics, and distributed tracing.
  • Propagate correlation identifiers across job, task, request, and output record, so any row can be traced to its source fetch.
  • Design artifact capture and retention so runs are reproducible and auditable.
  • Build dashboards, alerting, and runbooks that scale with the fleet.
  • Treat time‑to‑diagnosis as a tracked engineering metric.
• Service integration
  • Integrate the pipeline with authentication, relational, and analytical query services.
  • Apply timeouts, circuit breakers, bulkheads, and graceful degradation across service boundaries.
  • Design health checks and readiness signals that make system state explicit.
  • Build for partial‑dependency conditions, so the platform degrades predictably rather than unevenly.

Requirements

• 5 to 8 years building production Python systems, including async work with asyncio or equivalent.
• Deep hands‑on experience with Scrapy and Playwright or Selenium at scale.
• Proven work on authenticated portals. Session handling, cookie and token lifecycle, proxy rotation, and anti‑bot mitigation.
• Strong distributed queueing experience with RabbitMQ or Kafka. Consumer groups, redelivery, ordering, and dead‑letter queues.
• Hands‑on workflow orchestration experience. Airflow, Temporal, Prefect, Kestra, or a comparable system.
• Production experience with MongoDB and PostgreSQL, including index design and query tuning on large datasets.
• Practical observability experience. OpenTelemetry, Prometheus and Grafana or similar, structured logging, and distributed tracing.
• Solid grounding in HTTP and HTTPS, HTML, DOM, XPath, and CSS selectors.
• Comfortable with Linux, Docker, and CI/CD pipelines.
• Able to reason about a distributed failure across multiple services independently.

Nice to Have

• Trino, Presto, or comparable distributed query engines.
• Experience with e‑commerce vendor and seller portals.
• Data quality tooling such as Great Expectations or Soda, or a custom equivalent.
• Infrastructure as code.
• Failure injection or chaos testing practice.
• Experience mentoring engineers through code review and design review.

What We Are Looking For

• Ownership. You take responsibility for outcomes across system boundaries.
• Systems thinking. You look for the class of problem, not just the instance.
• Curiosity to analyse and reverse‑engineer websites and their defences.
• Evidence‑driven engineering. You reach for logs, traces, and data first.
• Clear written communication with data, analytics, and client‑facing teams.
• Comfort in a fast‑paced environment with shifting marketplace behaviour.

Company Description

1DigitalStack.ai is a Technology and Data Science product company helping brands win and grow profitably on e‑commerce marketplaces across the globe. Our platforms give customers deep e‑marketplace data, advanced and custom analytics, actionable intelligence, and end‑to‑end media optimization and automation. Brand Managers, P&L Owners, E‑commerce Managers, Channel and Category Managers, and Marketing Leaders use our solutions to unlock new revenue opportunities every day. We partner with some of India's largest consumer brands, including Unilever, Marico, Coke, Unicharm, Tata Consumer, and Dabur. We operate across 220+ global marketplaces spanning Southeast Asia, Europe, and the UAE.