Databricks Engineer/Lead

Impetus Technologies; New Delhi, Pune District, Bengaluru; On-site; INR 1,800,000 - 3,000,000; Full time

Job summary

Impetus Technologies in Delhi is seeking a seasoned Data Engineer to design and build scalable data pipelines using Databricks, Spark, PySpark, SQL and Delta Lake. You will implement bronze/silver/gold layers and support data quality, lineage and conformance.

You'll work on enterprise-grade batch and streaming pipelines, optimize Spark jobs, handle large historical datasets and contribute to documentation and knowledge transfer. Strong experience in investment management data domain is a plus.

Qualifications

• 7+ years in data engineering or similar roles.
• Proficient with Databricks, Spark, PySpark, SQL and Delta Lake.
• Experience in enterprise-scale batch and/or streaming pipelines.
• Strong knowledge of medallion architecture and data quality.

Responsibilities

• Develop scalable pipelines using Databricks, Spark, PySpark and SQL.
• Implement bronze/silver/gold pipelines per architecture.
• Handle ingestion, cleansing, transformation and conformance.
• Build reusable pipeline components for multiple sources.
• Support backward- and forward-looking historical processing.
• Optimize Spark jobs, Delta tables and SQL workloads.
• Create tests, SIT/UAT support and documentation.

Skills

Data engineering

Batch and streaming pipelines

Databricks

Apache Spark

PySpark

SQL

Delta Lake

Data quality

Schema evolution

Debugging & performance

Education

Bachelor's or Master's in CS/Engineering or related

Tools

Databricks

Apache Spark

PySpark

SQL

Delta Lake

Job description

• Develop scalable pipelines using Databricks, Apache Spark, PySpark, SQL and Delta Lake.
• Build Bronze, Silver and Gold pipelines following the approved architecture.
• Implement source ingestion, cleansing, transformation, enrichment and conformance.
• Develop reusable pipeline components for multiple sources.
• Implement incremental, historical and schema-evolution processing.
• Build portfolio accounting/custody data products covering holdings, tax lots, transactions, GL, cash, accruals and corporate actions.
• Process benchmark/index data and integrate Security Master and Account Master data.
• Implement enterprise identifier matching and cross-references.
• Implement data-quality validation, reconciliation, exception handling and certification.
• Support up to 10 years of historical backfill with reconciliation and auditability.
• Optimize Spark jobs, Delta tables and SQL workloads.
• Develop unit/integration tests and support SIT, UAT, production deployment and stabilization.
• Contribute to documentation and knowledge transfer.

Required Experience

• 712+ years in data engineering.
• 4+ years hands-on Databricks/Spark experience.
• Strong Databricks, Apache Spark, PySpark, SQL and Delta Lake skills.
• Enterprise-scale batch and/or streaming pipeline experience.
• Strong medallion architecture knowledge.
• Data transformation, cleansing, enrichment and conformance experience.
• Data-quality and reconciliation framework experience.
• Large historical dataset experience.
• Strong debugging and performance-tuning skills.

Financial Services Experience

Strong preference for investment management, asset management, portfolio accounting, custody, ABOR/IBOR, securities, benchmarks/indexes, Security Master or Account Master.

Preferred Databricks Skills

Databricks Jobs/Workflows, Lakeflow/declarative pipelines, Unity Catalog, Databricks SQL, Delta optimization, schema evolution, SCD, lineage, data-quality frameworks, metadata-driven ingestion and large-scale historical backfill.

Out of Scope

Azure infrastructure, VNet/Private Link/networking, Terraform, Azure landing zone, CI/CD platform engineering, Databricks workspace provisioning and infrastructure operations.

Education

Bachelor's or Master's degree in Computer Science, Engineering, Information Technology, Data Engineering or related discipline.