Data Engineer

Elfonze Technologies; Hyderabad; On-site; INR 2,000,000 - 4,000,000; Full time

Job summary

Elfonze Technologies is seeking a Senior Data Engineer with 5-8 years of experience to design and optimize data pipelines using Azure Databricks, Delta Lake, and Lakehouse architecture. The role emphasizes real-time ingestion, scalable batch/streaming processes, and governance.

You will collaborate on building modular Python frameworks, ensuring data quality and cost-effective processing, while leveraging Spark, PySpark, and SQL to transform and optimize data across Bronze-Silver-Gold layers.

Qualifications

• 5-8 years of experience in data engineering and lakehouse architectures.
• Strong hands-on with Databricks Lakehouse Platform.
• Experience with real-time data ingestion and streaming.
• Proficiency in Python, PySpark, and SQL for data pipelines.
• Ability to design modular, metadata-driven frameworks.
• Knowledge of data governance and quality frameworks.

Skills

Data engineering

Python

SQL

Spark

Lakehouse

Tools

Azure Databricks

Delta Lake

Unity Catalog

Databricks Apps

Databricks Workflows

Databricks Asset Bundles

Azure Event Hubs

Structured Streaming

Auto Loader

Lakebase

PostgreSQL

Job description

5-8 years exp

Azure Databricks: Strong hands-on experience with Databricks LakehouseArchitecture, Delta Lake, Unity Catalog, Databricks Apps, Workflows/Jobs, andDatabricks Asset Bundles (DAB).

Real-Time Data Ingestion: Experience with Azure Event Hubs, Spark StructuredStreaming, Auto Loader, Databricks Zerobus Ingest, and Delta Change Data Feed(CDF).

Python & SQL: Strong proficiency in Python, PySpark, and SQL fordata engineering, transformation, optimization, and automation.

Configuration-Driven Framework: Experience designing reusable, modular,metadata-driven Python frameworks with configurable business rules, executionengines, and orchestration layers.

Lakebase (PostgreSQL): Hands-on experience with Lakebase for operationaldata storage, relational data modeling, SQL development, and Lakebase-to-Deltasynchronization.

Databricks Lakehouse: Experience implementing Bronze, Silver, and Goldarchitectures, Delta tables, data transformations, and scalable analyticalpipelines.

Data Pipeline Engineering: Ability to design batch and streamingpipelines with error handling, retries, replay, reconciliation, and failurerecovery mechanisms.

Data Quality & Governance: Experience implementing validationframeworks, data quality checks, quarantine handling, audit logging, and UnityCatalog governance.

Data Modeling & Historical Processing: Strong understanding ofrelational data modeling, source-to-target mappings, CDC, SCD Type 2, andhistorical data synchronization.

Performance Optimization: Ability to optimize Spark workloads, SQLqueries, Delta tables, job execution, and data processing costs.