Python Engineer

Experiture Global

Kolkata

On-site

INR 1,200,000 - 1,800,000

Full time

11 days ago

Get a reply from this employer

Your resume and cover letter, tailored to exactly what they’re hiring for.

Get past ATS filters

Job summary

Experiture Global is seeking a hands-on Staff Engineer to own the Customer Data Platform (CDP). You will design and build data pipelines that turn customer records, behavioral events, and campaign activity into reliable data for segmentation, personalization, and campaigns.

This role combines Python engineering with lakehouse architecture, data quality, and production reliability, using Python, Polars, Delta Lake, PostgreSQL, and Azure.

Qualifications

  • 4+ years of professional Python experience with production data pipelines or data platforms.
  • Demonstrated ability to lead architecture decisions and deliver complex data engineering initiatives from design through production operation.
  • Strong experience building ETL/ELT pipelines, including incremental processing, deduplication, schema evolution, and recovery from partial failures.
  • Hands-on experience with Polars, Spark/PySpark, or a comparable data processing framework, and the ability to optimize processing for memory and throughput.
  • Experience with Delta Lake or a comparable lakehouse table format, including transactional writes, merges, and schema management.
  • Strong SQL and PostgreSQL skills, including data modeling, indexing, query optimization, and semi-structured data.
  • Experience building Python APIs and asynchronous background services, preferably with FastAPI.
  • Experience with message queues or event streaming systems, such as Azure Service Bus, Azure Event Hubs, or comparable services.
  • Practical experience with cloud object storage and cloud data infrastructure. Azure experience is preferred; strong AWS or GCP experience will be considered.
  • Strong understanding of data quality, automated testing, monitoring, and production incident diagnosis.
  • Experience designing secure, multi-tenant systems with appropriate access controls and protection of sensitive customer data.
  • Familiarity with Docker, GitHub Actions, and modern CI/CD workflows.

Responsibilities

  • Own and evolve the CDP's data engineering architecture, from source ingestion through curated datasets and downstream consumption.
  • Design, build, and optimize batch and event-driven pipelines for customer records, behavioral events, and communication activity.
  • Develop efficient transformations using Polars and Delta Lake, with Spark/Databricks where appropriate for larger workloads.
  • Implement schema-driven ingestion, field mapping, type normalization, validation, deduplication, and incremental upserts.
  • Evolve Bronze, Silver, and Gold data layers, including schema evolution, partitioning, and reliable merge strategies.
  • Build data profiling and quality controls that surface missing, invalid, inconsistent, or stale data.
  • Develop datasets and processing workflows for customer segmentation, audience materialization, personalization, and analytics.
  • Build and maintain FastAPI services for ingestion orchestration, schema management, job tracking, and data access.
  • Engineer reliable background workers and messaging workflows, including retries, idempotency, failure recovery, and dead-letter handling.
  • Optimize pipeline throughput, memory usage, query performance, and cloud operating costs.
  • Own pipeline observability, automated validation, and production troubleshooting while maintaining tenant isolation and secure handling of customer data.
  • Set engineering standards, mentor engineers, and collaborate with product, application, and ML teams on data contracts and platform capabilities.

Skills

Python
Data pipelines
Data engineering
SQL
Polars
Delta Lake
Spark/PySpark
FastAPI
Azure
AWS/GCP

Tools

Spark/PySpark
Delta Lake
PostgreSQL
FastAPI
Azure
Docker
GitHub Actions
Azure Service Bus

Job description

Staff Engineer, Customer Data Platform


Were seeking a hands-on, architect-level Data Engineer to take technical ownership of the customer data platform (CDP) supporting our marketing, analytics, and AI-driven applications. Youll design and build the pipelines that turn customer records, behavioral events, and campaign activity into reliable, usable data for segmentation, personalization, and campaign automation.

This role combines strong Python engineering with deep expertise in data processing, lakehouse architecture, and production reliability. Youll own the platform's ingestion, transformation, schema management, data quality, and serving capabilities, working primarily with Python, Polars, Delta Lake, PostgreSQL, and Azure.


Key Responsibilities:

  • Own and evolve the CDP's data engineering architecture, from source ingestion through curated datasets and downstream consumption.
  • Design, build, and optimize batch and event-driven pipelines for customer records, behavioral events, and communication activity.
  • Develop efficient transformations using Polars and Delta Lake, with Spark/Databricks where appropriate for larger workloads.
  • Implement schema-driven ingestion, field mapping, type normalization, validation, deduplication, and incremental upserts.
  • Evolve Bronze, Silver, and Gold data layers, including schema evolution, partitioning, and reliable merge strategies.
  • Build data profiling and quality controls that surface missing, invalid, inconsistent, or stale data.
  • Develop datasets and processing workflows for customer segmentation, audience materialization, personalization, and analytics.
  • Build and maintain FastAPI services for ingestion orchestration, schema management, job tracking, and data access.
  • Engineer reliable background workers and messaging workflows, including retries, idempotency, failure recovery, and dead-letter handling.
  • Optimize pipeline throughput, memory usage, query performance, and cloud operating costs.
  • Own pipeline observability, automated validation, and production troubleshooting while maintaining tenant isolation and secure handling of customer data.
  • Set engineering standards, mentor engineers, and collaborate with product, application, and ML teams on data contracts and platform capabilities.

Required Qualifications:

  • 4+ years of professional Python experience, including substantial ownership of production data pipelines or data platforms.
  • Demonstrated ability to lead architecture decisions and deliver complex data engineering initiatives from design through production operation.
  • Strong experience building ETL/ELT pipelines, including incremental processing, deduplication, schema evolution, and recovery from partial failures.
  • Hands-on experience with Polars, Spark/PySpark, or a comparable data processing framework, and the ability to optimize processing for memory and throughput.
  • Experience with Delta Lake or a comparable lakehouse table format, including transactional writes, merges, and schema management.
  • Strong SQL and PostgreSQL skills, including data modeling, indexing, query optimization, and semi-structured data.
  • Experience building Python APIs and asynchronous background services, preferably with FastAPI.
  • Experience with message queues or event streaming systems, such as Azure Service Bus, Azure Event Hubs, or comparable services.
  • Practical experience with cloud object storage and cloud data infrastructure. Azure experience is preferred; strong AWS or GCP experience will be considered.
  • Strong understanding of data quality, automated testing, monitoring, and production incident diagnosis.
  • Experience designing secure, multi-tenant systems with appropriate access controls and protection of sensitive customer data.
  • Familiarity with Docker, GitHub Actions, and modern CI/CD workflows.

Preferred Qualifications:

  • Experience building customer data platforms or systems for customer profiles, behavioral events, segmentation, or marketing automation.
  • Hands-on experience with Azure Data Lake Storage Gen2, Azure Service Bus, Azure Event Hubs, and Databricks.
  • Experience using Polars lazy execution, PyArrow, Parquet, and delta-rs to process datasets efficiently.
  • Familiarity with customer identity resolution, consent and preference data, and data governance.
  • Experience preparing reliable datasets for downstream ML and AI applications.
Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Senior Software Engineer - Data Platform Engineering
Senior Software Engineer - Data Platform Engineering

Myntra • Bengaluru

On-site
INR 1,200,000 - 1,600,000
Senior Data Engineer
Senior Data Engineer

QUESTK2 Technologies • Hyderabad

Hybrid
INR 1,400,000 - 2,300,000
Python Data Engineer
Python Data Engineer

Moptra • Bengaluru

Remote
INR 1,500,000 - 2,500,000
Python Developer
Python Developer

Gemini Solutions • Gurugram District

On-site
INR 1,200,000 - 1,800,000
Senior Data Engineer
Senior Data Engineer

SOTIIND SOTI India Private Ltd. • Ernakulam

On-site
INR 1,500,000 - 2,000,000
Data Engineer
Data Engineer

InfoVision Inc. • Pune District

On-site
INR 4,000,000 - 6,000,000
Lead Data Engineer
Lead Data Engineer

Algoworks Technologies Private Limited • Dadri

On-site
INR 1,500,000 - 2,100,000
Data Engineer
Data Engineer

Recro • Bengaluru

On-site
INR 1,200,000 - 2,400,000
Data Engineer
Data Engineer

Valuenode Private Limited • India

Remote
INR 1,200,000 - 1,800,000
Senior Data Engineer
Senior Data Engineer

Proclink • Gandhamguda

On-site
INR 800,000 - 1,500,000