Data Engineer

Clovertex; Hyderabad; On-site; INR 2,500,000 - 4,000,000; Full time

Job summary

Keka Inc. in Hyderabad is seeking a Data Engineer with strong Databricks experience to build and integrate clinical, biomarker and multi-omics datasets into a governed, analytics-ready data platform.

The role focuses on scalable data pipelines using PySpark, Python and SQL, Delta Lake, data quality and lineage, and collaboration with the Data Architect and Bioinformatics SME on domain models for analytics and AI use cases across research pipelines.

Qualifications

• 4+ years of Data Engineering experience.
• Hands-on Databricks, PySpark, Python and SQL expertise.
• Spark, Delta Lake and Medallion Architecture.
• ETL/ELT and data-pipeline development.
• Data modelling and data-quality implementation.
• Cloud object storage, preferably AWS S3.
• Git and CI/CD experience.

Responsibilities

• Develop ingestion and transformation pipelines using PySpark, Python and SQL.
• Build and maintain Delta Lake tables and analytical datasets.
• Integrate and harmonize structured and semi-structured scientific datasets.
• Implement data-quality, validation and reconciliation rules.
• Develop incremental loading, exception handling and reprocessing mechanisms.
• Support Unity Catalog, RBAC, metadata and data lineage.
• Work with the Data Architect and Bioinformatics SME on domain-specific transformations and data models.
• Develop Gold-layer curated datasets for analytics and downstream AI use cases.
• Support testing, UAT, documentation and production readiness.

Skills

Databricks

PySpark

Python

SQL

Delta Lake

Medallion Architecture

ETL/ELT

Data modelling

Data quality

AWS S3

Git

CI/CD

Unity Catalog

RBAC

Tools

Unity Catalog

RBAC

Job description

Role Summary

We are looking for a Data Engineer with strong Databricks experience to build and integrate clinical, biomarker and multi-omics datasets into a governed, analysis-ready data platform. The candidate should have exposure to Bioinformatics/Life Sciences data and experience developing scalable data pipelines on Databricks.

Key Responsibilities

• Develop ingestion and transformation pipelines using PySpark, Python and SQL.
• Build and maintain Delta Lake tables and analytical datasets.
• Integrate and harmonize structured and semi-structured scientific datasets.
• Implement data-quality, validation and reconciliation rules.
• Develop incremental loading, exception handling and reprocessing mechanisms.
• Support Unity Catalog, RBAC, metadata and data lineage.
• Work with the Data Architect and Bioinformatics SME on domain-specific transformations and data models.
• Develop Gold-layer curated datasets for analytics and downstream AI use cases.
• Support testing, UAT, documentation and production readiness.

Must Have

• 4+ years of Data Engineering experience.
• Strong hands-on Databricks, PySpark, Python and SQL.
• Spark, Delta Lake and Medallion Architecture.
• ETL/ELT and data-pipeline development.
• Data modelling and data-quality implementation.
• Cloud object storage, preferably AWS S3.
• Git and CI/CD experience.

Preferred

• Experience with genomics, biomarker or multi-omics datasets.
• Exposure to RNA-seq, WES/WGS, VCF, CNV or similar scientific datasets.
• Unity Catalog and Databricks governance.