Software Engineer II (Data Engineering)

ZAGENO Inc.; Bengaluru; In loco; INR 1.200.000 - 1.800.000; Tempo pieno

Descrizione del lavoro

ZAGENO Inc. in Bengaluru is seeking a Data Engineer to own the operational data pipelines powering our life sciences catalog. You’ll ensure reliability, correctness, and scalability, partnering withData Science and Analytics to deliver clean data.

You’ll also drive observability, testing, and incident response for fast, trusted data. The role requires 3+ years in data engineering, strong Python and SQL, and experience with Databricks, Delta Lake, Airflow, and CDC patterns.

Competenze

• 3+ years as a Data Engineer owning production pipelines.
• Strong Python data engineering and SQL skills.
• Experience with Databricks, Delta Lake, and Airflow.
• Experience implementing CDC patterns for data sync.
• Experience building low-latency APIs for ops/analytics.
• Test-driven development with unit and integration tests.
• Independent work style under ambiguity; durable architectures.

Mansioni

• Own reliability and performance of operational pipelines.
• Reduce technical debt with designed, testable logic.
• Build low-latency data APIs for downstream use.
• Implement CDC to keep catalogs in sync.
• Develop monitoring, observability, and automated tests.
• Design master data logic at catalog scale.
• Translate non-technical requirements into durable pipelines.
• Own code versioning, deployment, and incident response.
• Leverage BigQuery, Databricks, Spark/PySpark, AWS/GCP.

Conoscenze

Python

SQL

Databricks

Airflow

CDC patterns

APIs

Testing

Independence

Strumenti

Delta Lake

Spark/PySpark

Descrizione del lavoro

About the Role

ZAGENO is hiring a Data Engineer to own the operational data pipelines powering our life sciences catalog. This is a high-ownership role: you’ll be the primary engineer responsible for the reliability, correctness, and scalability of the pipelines our operations teams depend on daily.

The work spans production pipeline reliability, reducing accumulated technical debt in existing systems, building observability and testing infrastructure, and partnering with Data Science and Analytics on clean data delivery. You’ll work directly with CatalogOps and business stakeholders – turning evolving, often underspecified business rules into architectures that stay flexible without compromising data quality.

You’ll make architectural decisions, push back on requests that introduce heuristic debt, and own incident response end-to-end.

In this role you will:

• Own reliability and performance of operational pipelines across our product catalog infrastructure.
• Identify and reduce technical debt, replacing reactive patches with designed, testable logic.
• Build and maintain low latency data APIs that serve downstream operational and analytics consumers.
• Implement CDC patterns to keep catalog data synchronized across systems with minimal lag.
• Build monitoring, observability, and automated testing so failures surface before stakeholders report them.
• Design and implement unit standardization and master data logic at catalog scale.
• Translate business requirements from non-technical stakeholders into durable pipeline logic.
• Own code versioning, deployment, and incident response for your layer.
• Leverage your expertise in the tech stack: BigQuery, Databricks, Spark/PySpark, AWS/GCP.

About you

Required

• 3+ years as a Data Engineer, including solo or primary ownership of production pipelines
• Strong Python – data engineering, transformation logic, testing discipline
• Strong SQL with ability to write correct queries, identify and refactor anti-patterns
• Databricks, Delta Lake, Airflow for production orchestration
• Experience with CDC patterns for real-time or near-real-time data synchronization
• Experience building low latency APIs serving operational or analytical consumers
• Test‑driven development discipline – unit tests, integration tests, regression coverage as standard practice, not afterthought
• Operates independently under ambiguity; designs systems to be maintained, not just to run

Preferred

• Kafka or equivalent event streaming platform experience
• Experience with entity matching, deduplication, or master data management
• Exposure to ML pipeline support in production
• Familiarity with NLP techniques for entity resolution or text normalization (tokenization, similarity matching, named entity recognition)

What success looks like

• Engineers ship data products: outputs are documented, versioned, and designed for reuse across Analytics, Data Science, and operational consumers.
• Reliability: pipelines run reliably with minimal manual intervention.
• Performance: data latency and downtime decrease measurably over time.
• Data quality: Analytics and Data Science teams receive clean, trustworthy data without ad hoc fixes.
• Scalability: infrastructure scales with volume growth without proportional cost increase.
• Reduction of debt: technical debt in existing pipelines decreases measurably as heuristic patches are replaced with designed logic.
• Synchronization: CDC‑driven synchronization eliminates the class of cross‑pipeline identity drift bugs.