Senior Data Engineer

ShimentoX Technologies; Bengaluru; On-site; INR 4,000,000 - 7,000,000; Full time

Job summary

ShimentoX Technologies is seeking a Senior Data Engineer in Bengaluru to design, build, and own end-to-end production data pipelines powering analytics and decision-making. You will work with Python and SQL, focusing on robust data architectures for finance, risk, or compliance domains.

The role emphasizes data quality, auditability, lineage, and scalable medallion architecture, with cross-dialect SQL, idempotent reload patterns, and Airflow-style orchestration.

Qualifications

• 6+ years of data engineering experience.
• Strong data architecture and design skills.
• End-to-end production ownership of pipelines.
• Expert Python and SQL skills; Hive/Hadoop is a nice-to-have.
• Robust data quality and auditability practices.
• Experience with finance, risk, or compliance data.

Responsibilities

• Own data architecture decisions for pipelines and models.
• Own production pipelines from design to operation.
• Build transactional fact tables and medallion models.
• Support finance/risk/compliance use cases with data lineage.
• Implement data quality and auditability controls.
• Write complex cross-dialect SQL (MySQL + PostgreSQL).
• Design idempotent reload patterns and templated SQL.
• Orchestrate DAGs with cross-pipeline dependencies.
• Develop Python ETL tooling with pandas and SQLAlchemy.
• Integrate with Google Drive/Sheets APIs and YAML configurations.

Skills

Data engineering

Data architecture

End-to-end ownership

Python

SQL

Data quality

Auditability

Pandas

SQLAlchemy

CI/CD

Git workflows

ETL tooling

Airflow

Cross-dialect SQL

Tools

MySQL

PostgreSQL

Airflow

Jinja

Git

Docker

Job description

We're looking for a Senior Data Engineer with strong data architecture and system design skills to design, build, and own end-to-end production pipelines powering analytics and decision-making across the organization. The role centers on deep Python and SQL expertise, robust transaction-level fact tables, and rigorous data quality and auditability, ideally within a finance, risk, or compliance data domain.

What You'll Do

• Own data architecture and system design decisions for pipelines and data models, including grain, partitioning, schema evolution, and scalability trade-offs.
• Take end-to-end ownership of production pipelines: design, build, deploy, monitor, and operate.
• Design and build transaction tables (append-only, immutable event/transaction-grain fact tables) alongside standard fact/dimension models using medallion (bronze/silver/gold) architecture.
• Build and support pipelines for finance, risk, or compliance use cases where accuracy, auditability, and data lineage are critical.
• Implement data quality and auditability controls, including validation checks, reconciliation logic, anomaly detection, and audit trails for every pipeline.
• Write complex, cross-dialect SQL (MySQL + PostgreSQL), including window functions, multi-layered CTEs, CASE-driven logic, COALESCE/NULLIF, type casting, and date/timestamp handling.
• Build idempotent reload patterns (DELETE + INSERT), UNION ALL/set operations, and templated (Jinja-style) SQL, including handling late-arriving/corrected transaction records without double-counting.
• Orchestrate Airflow-style DAGs with cross-pipeline dependencies and secure credential handling.
• Build Python ETL tooling using pandas (CSV DB), SQLAlchemy + raw drivers (MySQLdb, psycopg2), type hints/dataclasses, and class-based pipeline design.
• Integrate with Google Drive/Sheets APIs, build YAML-driven pipeline configurations, and handle pipeline staging requirements.
• Own CI/CD for pipelines through standard Git workflows.
• Document data models and pipeline logic for maintainability, keeping compliance and audit requirements in mind.

What We're Looking For

Must-Have Skills:

• 6+ years of experience in data engineering.
• Strong data architecture and system design skills to make sound trade-offs on modeling, partitioning, and scalability—not just write queries.
• Demonstrated end-to-end production ownership of data pipelines from design through deployment, monitoring, and operational support.
• Expert-level Python and SQL as the primary toolset. Hive/Hadoop is a nice-to-have only.
• Robust data quality and auditability practices, including validation, reconciliation, anomaly detection, and audit trails.
• Experience working with finance, risk, or compliance data, with an understanding of accuracy, auditability, and data-lineage requirements.
• Proven experience designing and building transaction/fact tables.
• Expert SQL skills, including window functions, CTEs, CASE logic, and cross-dialect MySQL/PostgreSQL.
• Strong Python skills with pandas, SQLAlchemy, type hints/dataclasses, and class-based pipelines.
• Solid data modeling fundamentals, including fact/dimension models, normalization, and medallion architecture.
• YAML-based configuration experience, along with standard CI/CD and Git workflows.

Nice-to-Have Skills