Senior Data Engineer

Zemoso Technologies; Bengaluru Urban; Presencial; INR 3.500.000 - 7.000.000; Jornada completa

Descripción de la vacante

Zemoso Technologies is seeking a Senior Data Engineer to provide technical leadership and hands-on ownership across client engagements.

The role sits between Data Engineer and Principal Engineer, shaping cross-project standards, accelerators, and end-to-end delivery of complex data platforms.

You will design, build, and optimize ETL/ELT pipelines, mentor engineers, and ensure production readiness with scalable, observable solutions.

Formación

• Proficient in Python and SQL with hands-on Spark expertise.
• Experience building production-grade ETL/ELT pipelines and orchestration.
• Cloud platform experience across AWS/Azure/GCP and data modeling knowledge.
• Strong leadership and mentorship capabilities.

Responsabilidades

• Lead design for complex data subsystems including ingestion, transformation, and serving.
• Define standard DAGs and reusable templates to accelerate delivery.
• Architect orchestration patterns with Airflow or equivalents.
• Mentor senior and mid-level engineers and review designs.
• Produce architecture docs and ADRs with trade-offs and risks.

Conocimientos

Python

SQL

Spark / PySpark / Databricks

Data modeling

Data warehouses

Data lakes

Cloud platforms (AWS / Azure / GCP)

Lead technical decisions

Mentor engineers

Design docs / ADRs

Herramientas

Airflow

ADR documentation

Descripción del empleo

We are looking for a Senior Data Engineer who provides technical leadership and hands-on ownership for complex, high-impact data engineering solutions across client engagements.

This role sits between Data Engineer and Principal Engineer, extending senior execution into cross-project design decisions, standards, and accelerators, while still delivering critical components end-to-end.

The Principal Data Engineer reduces ambiguity, drives technical direction, and raises engineering quality across teams.

Key Responsibilities

Technical Leadership & Solution Design

• Lead design for complex data subsystems (ingestion, transformation, curation, serving).
• Drive key design choices such as batch vs streaming, lakehouse vs warehouse, orchestration approach, and partitioning strategy.
• Define and enforce standards for pipeline design, coding practices, documentation, and production readiness.
• Design, build, and optimize ETL / ELT pipelines using Python, SQL, and distributed processing frameworks (Spark / Databricks).
• Own complex integrations across multiple source systems (APIs, files, CDC/event streams).
• Optimize pipelines for performance, reliability, and cost efficiency.

Orchestration, Reliability & Operations

• Architect and implement orchestration patterns(Airflow or equivalent), including retries, backfills, SLAs, and dependency management.
• Establish monitoring, alerting, and data quality checks.
• Lead triage and resolution of high-severity data pipeline issues.

Platform Enablement (Applied, Not Architectural-Heavy)

• Build reusable components and templates (ingestion patterns, validation utilities, standard DAGs) to accelerate delivery.
• Partner with Architects to align solutions with modern cloud data platform patterns where needed.
• Mentor Senior and mid-level engineers through design and code reviews.
• Clearly communicate design decisions and trade-offs to engineers, product teams, and clients/stakeholder.
• Produce and maintain design docs and Architecture Decision Records (ADRs) including trade-offs, risks, and rollout plans.
• Communicate design intent and responsibilities across teams clearly.

Must-Have Skills

• Strong proficiency in Python and SQL.
• Deep hands-on experience with Spark / PySpark / Databricks.
• Solid experience designing production-grade ETL / ELT pipelines and orchestration workflows.
• Strong understanding of data modeling, data warehouses, and data lakes.
• Practical experience with cloud platforms (AWS / Azure / GCP).
• Proven ability to lead technical decisions, mentor engineers, and deliver under ambiguity.

Nice-to-Have Skills

• Experience with streaming or event-driven data pipelines (Kafka, Kinesis, Pub/Sub).
• Exposure to data observability, governance, or catalog tools.
• Familiarity with containers, IaC, or platform enablement practices.
• Relevant cloud or data engineering certifications.

What Success Looks Like in This Role

• You deliver the hardest and most ambiguous parts of client data platforms.
• You reduce ambiguity by turning requirements into clear designs and execution plans.
• Your pipelines are reliable, observable, cost-aware, and production-ready.
• You uplift the DE practice through standards, reusable components, and mentorship.