Senior Developer

HCL Technologies Limited; Ahmedabad District; On-site; INR 1,500,000 - 2,200,000; Full time

Job summary

HCLTech is seeking a Databricks Data Engineer to design and maintain scalable data pipelines for RCIS Crop Insurance data modernization. You will build Bronze/Silver data layers, implement metadata-driven ingestion, CDC and transformations, and ensure data quality and regulatory readiness across Policy and Claims domains.

You will collaborate with Data Architects, BI teams and Guidewire SMEs to deliver trusted datasets supporting analytics, dashboards and AI initiatives.

Qualifications

• Proficiency in Databricks, PySpark, Delta Lake and Spark SQL.
• Experience in building Bronze/Silver data layers and SSOT.
• Knowledge of data quality, reconciliation, lineage and audit controls.

Responsibilities

• Develop and optimize Databricks, PySpark and Delta Lake pipelines for batch and CDC-based ingestion from Guidewire CDA, CIMax, DB2 and external sources.
• Build Bronze/Silver layer data products and implement source-to-target mappings.
• Embed data quality, reconciliation, lineage, observability and audit controls in pipelines.

Skills

Databricks & Spark Development

Data Engineering & ETL/ELT

Data Modeling & Lakehouse

Data Quality & Reconciliation

Cloud & DevOps

Guidewire Data Expertise

Tools

Azure/AWS Databricks

Delta Lake

DLT (Delta Live Tables)

Job description

Databricks Data Engineer responsible for developing and maintaining scalable data pipelines that support the RCIS Crop Insurance Data Modernization program. Design and implement metadata-driven ingestion, CDC, transformation, and reconciliation frameworks to integrate Guidewire CDA, CIMax/DB2, reporting databases, and legacy systems into a unified Silver Layer data foundation. Build trusted Policy, Claims, Billing, APH Yield, Acreage, Premium, Subsidy, and Indemnity data products that establish a Single Source of Truth (SSOT). Collaborate with Data Architects, Data Analysts, Guidewire SMEs, and BI teams to deliver governed, audit-ready, high-quality datasets supporting regulatory reporting, analytics, dashboards, and future AI-driven capabilities.

Key Responsibilities

Develop and optimize Databricks, PySpark, and Delta Lake pipelines for batch and CDC-based ingestion from Guidewire CDA, CIMax, DB2, and external crop insurance data sources. Build Bronze and Silver layer data products, implement source-to-target mappings, and perform data transformations aligned to canonical Policy and Claims models. Embed data quality, reconciliation, lineage, observability, and audit controls within all pipelines. Support data profiling, testing, SIT/UAT validation, and report reconciliation activities. Develop automated monitoring, alerting, and performance optimization solutions. Collaborate with business SMEs and architects to ensure accurate, trusted, regulatory-compliant data supporting RCIS reporting and analytics initiatives.

Skill Requirements

Top 6 Key Skillsets – Databricks Data Engineer

1. Databricks & Spark Development
   • Azure/AWS Databricks
   • PySpark
   • Spark SQL
   • Delta Lake
   • Delta Live Tables (DLT)
   • Databricks Workflows
2. Data Engineering & ETL/ELT
   • Metadata-Driven Ingestion
   • CDC (Change Data Capture)
   • Batch & Incremental Processing
   • Data Transformation Frameworks
   • Source-to-Target Mapping (STTM)
3. Data Modeling & Lakehouse Architecture
   • Bronze/Silver Layer Development
   • Canonical Data Models
   • Medallion Architecture
   • Data Product Development
   • SSOT Implementation
4. Data Quality & Reconciliation
   • Data Profiling
   • Data Validation
   • Data Reconciliation
   • Audit Controls
   • Data Observability
   • Lineage & Traceability
5. Cloud & DevOps
   • AWS / Azure
   • ADF / AWS Glue
   • Git
   • CI/CD Pipelines
   • Airflow
   • Infrastructure Automation
6. Crop Insurance & Guidewire Data Expertise
   • Guidewire CDA (PC, CC, BC)
   • CIMax / DB2
   • Policy & Claims Data
   • APH Yield & Acreage Reporting
   • Premium, Subsidy & Indemnity Data
   • RMA Regulatory Reporting
   • Resume-Friendly Skill Summary
   • Databricks, PySpark, Delta Lake, ETL/ELT Development, Guidewire CDA Integration, Data Quality & Reconciliation, CDC Pipelines, Medallion Architecture, Data Governance, AWS/Azure Cloud, Crop Insurance Data Domains, Regulatory Reporting Support.

At HCLTech, you'll supercharge your potential. You'll find your career. And you'll find your spark. All at a place that knows that helping its customers stay on top starts by putting its people first.

HCLTech is a global technology company, home to more than 223,000 people across 60 countries, delivering industry-leading capabilities centered around digital, engineering, cloud and AI, powered by a broad portfolio of technology services and products. We work with clients across all major verticals, providing industry solutions for Financial Services, Manufacturing, Life Sciences and Healthcare, Technology and Services, Telecom and Media, Retail and CPG, and Public Services. Consolidated revenues as of 12 months ending June 2026totaled $14.8billion.