Data Engineer

SkillYT; Mumbai; On-site; INR 3,000,000 - 6,000,000; Full time

Job summary

SkillYT is seeking a data-substrate expert to own end-to-end batch data processing. You will design and operate ingestion pipelines, perform large-scale set-based SQL processing, and maintain strict tenant isolation and auditability. The role emphasizes a self-directed approach with strong testing, documentation, and reliability practices in a production Python/Django stack. Mumbai-based on-site role.

Qualifications

• 5–8 years of production systems experience
• Strong PostgreSQL: schema design, query planning, large-scale batch
• Ownership of ingestion, transformation, reconciliation, serving pipelines
• Strong SQL, set-based thinking, window functions, incremental computation
• Production Python backend experience and framework independence
• Test-driven development mindset with batch jobs and reconciliation systems

Responsibilities

• Own data substrate and batch runtime
• Build reliable ingestion pipelines from external data sources
• Use set-based SQL for large datasets and ensure nightly processing window SLAs
• Maintain tenant isolation and resilience against bad data
• Design backfill and replay capabilities as standard features
• Maintain immutable, auditable records and traceability of decisions
• Build for failures with retries, circuit breakers, and reconciliation mechanisms
• Develop strong alerting and monitoring and ensure idempotent jobs

Skills

Strong PostgreSQL expertise

Production Python backend

Strong SQL & data thinking

Strong written communication

High ownership

Production data pipeline ownership

Tools

PostgreSQL

Django

Python

Kafka

Job description

The Role

You own the data substrate and the batch runtime of a product we are building from scratch. This is a data-specialist seat, not a general backend seat. Routine application development is guided in weekly reviews; what cannot be substituted is your depth in data modeling, set-based computation, and pipeline correctness. You will work self-directed, writing your own specs and decisions, with your tests and documentation carrying the quality bar between reviews.

Stack: Python, PostgreSQL, Django, server-rendered frontend (htmx).

What You Will Own

• Deep third-party data ingestion
  • Build reliable ingestion pipelines for external data sources
  • Account for silently dropped webhooks through reconciliation with source-of-truth totals
• Large-scale batch computation
  • Use set-based SQL for processing large datasets
  • Ensure jobs complete within fixed nightly processing windows
• Multi-tenant reliability
  • Maintain strict tenant isolation
  • Ensure one tenant’s bad data does not impact another tenant’s processing
• Backfill & replay capabilities
  • Design systems to support backfills and replays as standard capabilities
  • Avoid relying on one-off emergency scripts
• Append-only & auditable data
  • Maintain immutable, auditable records
  • Ensure every automated decision can be reconstructed and traced
• Resilient vendor API integrations
  • Build for failures and unreliable external systems
  • Implement retries, circuit breakers and reconciliation mechanisms
• Operational reliability
  • Build strong alerting and monitoring
  • Handle database partitioning effectively
  • Design and maintain idempotent jobs

What We Are Looking For

Ideal Candidate Profile

• 5–8 years of experience building and operating production systems — years are a proxy; the skills below are the actual bar.
• Strong PostgreSQL expertise
  • Schema design
  • Query planning and optimization
  • Batch performance at real scale
• Production data pipeline ownership
  • Ingestion
  • Transformation
  • Reconciliation
  • Data serving
  • Experience should go beyond application CRUD/ORM work.
• Strong SQL & data thinking
  • Thinks in sets, not loops
  • Multi-step SQL transformations
  • Window functions
  • Statistical aggregates, percentiles & distributions
  • Incremental computation
  • Experience building cohort, retention or funnel metrics from raw event/order data
• Production Python backend experience
  • Experience with any Python backend framework is fine
  • Able to keep business logic cleanly separated from framework-specific code
• Strong testing mindset
  • Tests are part of the development process, not an afterthought
  • Especially experienced with testing batch jobs and reconciliation systems where data errors can remain unnoticed
• Production problem-solving / "war stories"
  • Has dealt with real incidents such as:
  • Queues backing up
  • Webhooks silently dropping
  • Data loss or inconsistencies
  • Reconciliation catching issues that went unnoticed
• Strong written communication
  • Comfortable with documentation-driven development
  • Can write clear specs, decisions and technical documentation
• High ownership
  • Comfortable building systems from scratch
  • Can independently own outcomes with minimal supervision

Good to have

• Production Django experience
• E-commerce / D2C platform experience
• Streaming & event-driven architecture
  • Kafka or similar systems
• Familiarity with probabilistic data structures
  • HyperLogLog
  • Bloom Filters
  • t-digest
• Comfort with server-rendered applications/screens when required

Skills: postgresql,django,python