Senior Data Engineer (Python, Pandas & Numpy)

Lumiq; Dadri; Presencial; INR 1 400 000 - 2 300 000; Tempo integral

Resumo da oferta

Lumiq is seeking a Senior Data Engineer (Python, Pandas & NumPy) to re-engineer our ETL pipelines, migrating from Microsoft SQL Server to PostgreSQL/CURA while preserving load patterns into Elasticsearch and MongoDB.

You will design efficient SQL, containerize ETLs with Docker and Kubernetes, ensure data integrity and performance, and collaborate with DevOps to deploy in production; prior experience with mission-critical systems is required.

Qualificações

• Extensive experience building large-scale distributed data systems.
• Proficient in Python with pandas, NumPy, and Polars.
• Advanced SQL development and performance tuning.
• Experience with Docker and Kubernetes in production.
• Familiar with Elasticsearch and MongoDB data stores.
• Experience with ETL pipelines and data migration.

Responsabilidades

• Analyze existing ETL pipelines and dependencies on MS SQL Server.
• Design modifications to repoint ETL extractions to PostgreSQL (CURA) while preserving transformations and loads to Elasticsearch and MongoDB.
• Ensure end-to-end data integrity, quality, and freshness after source switch.
• Write efficient SQL queries to extract data from the new source.
• Conduct performance testing to ensure no degradation of pipeline throughput or latency.
• Containerize, orchestrate, and deploy updated ETLs using Docker and Kubernetes.
• Monitor post-deployment performance and resolve production issues.
• Document design, code, data mappings, and runbooks.

Conhecimentos

Python

Pandas

NumPy

SQL

Docker

Kubernetes

Elasticsearch

MongoDB

ETL

Data pipelines

Ferramentas

PostgreSQL

Docker

Kubernetes

Descrição da oferta de emprego

About the job Senior Data Engineer (Python, Pandas & Numpy)

Job Purpose :
We are seeking a highly skilled ETL Data Engineer to re-engineer our existing data pipelines to extract data from a new data source (PostgreSQL / CURA system) instead of the current Microsoft SQL Server (CRM persistence store), while preserving the existing load patterns to Elasticsearch and MongoDB. The engineer will ensure this migration has zero impact on data quality, system performance, or end-user experience.

Key Responsibilities :

• Analyze existing ETL pipelines and their dependencies on Microsoft SQL Server as source systems.
• Design and implement modifications to repoint ETL extractions from PostgreSQL (CURA) while preserving the current transformations and load logic into Elasticsearch and MongoDB.
• Ensure end-to-end data integrity, quality, and freshness remain unaffected after the source switch.
• Write efficient and optimized SQL queries to extract data from the new source.
• Conduct performance testing to confirm no degradation of pipeline throughput or latency in production.
• Work closely with DevOps and platform teams to containerize, orchestrate, and deploy the updated ETLs using Docker and Kubernetes.
• Monitor post-deployment performance and handle any production issues proactively.
• Document design, code, data mappings, and operational runbooks.

Required Skills and Qualifications :

• Strong experience building and maintaining large-scale distributed data systems.
• Expert-level proficiency in Python, especially data analysis/manipulation libraries like pandas, NumPy, and Polars.
• Advanced SQL development skills with proven experience in performance optimization.
• Working knowledge of Docker and Kubernetes.
• Familiarity with Elasticsearch and MongoDB as data stores.
• Experience working in production environments with mission-critical systems.