Senior Data Engineer

Intelaltus Llp; Gurugram District; Hybrid; INR 3,500,000 - 6,000,000; Full time

Job summary

Intelaltus Llp in Gurugram is seeking an experienced Senior Data Engineer to design, develop and operate enterprise-scale data platforms using Azure Databricks. You will own end-to-end data pipelines, ensure data governance, quality, and security, and collaborate with Product, Engineering, AI and Infrastructure teams to deliver reliable data solutions for financial services.

This role requires 10+ years of data engineering and hands-on expertise with SFTP ingestion, Delta Lake, Databricks

Qualifications

• 10+ years of data engineering experience with end-to-end ownership.
• Proficiency in Azure Databricks, Spark and Delta Lake.
• Strong SQL and Python skills.
• Experience with data governance, security and PII handling.
• Experience building secure, auditable data pipelines.

Responsibilities

• Lead end-to-end data pipelines, including secure file ingestion (SFTP), landing, historical storage (Delta) and curated data layers.
• Own Azure Databricks delivery, including notebooks, Workflows/Jobs, Asset Bundles, parameters, concurrency and retries.
• Implement medallion architecture (Landing Staging Curated Analytics) with partitioning, checksums and metadata sidecars.
• Maintain end-to-end pipeline auditability using correlation IDs, trigger records and completion records.
• Partner with SRE teams on data lake/SFTP storage, identity management and secrets for Databricks and related services.
• Implement data quality controls, schema evolution, lineage, cost optimization and runbooks for pipeline incidents.
• Provide curated, quality-assured datasets from data lakes and SQL warehouses to Product and AI teams for RAG and embedding workflows.
• Ensure data contracts cover schema, metadata, access classifications and refresh frequency. Ownership of RAG corpora and vector stores remains with the respective Product and AI teams.

Skills

Azure Databricks

Spark

Delta Lake

SQL

Python

Data governance

RBAC & security

PII handling

Data pipelines

Tools

SFTP ingestion

CI/CD for data engineering

Workflows/Jobs

Asset Bundles

Delta Live Tables

dbt

Event Hubs

CDC

Job description

We are looking for an experienced Senior Data Engineer to lead the design, development and operation of enterprise-scale data platforms using Azure Databricks.

The role involves building secure, scalable and auditable data pipelines, managing large-scale data platforms and ensuring high standards of data governance, quality and operational excellence.

The position also involves collaborating with Product, Engineering, AI and Infrastructure teams to deliver reliable data solutions for global financial services environments.

Experience: 10+ Years Location: Gurugram Work Model: Hybrid 2 Days Work from Office Preference: Candidates comfortable with a hybrid work arrangement are strongly preferred.

Key Responsibilities

• Lead end-to-end data pipelines, including secure file ingestion (SFTP), landing, historical storage (Delta) and curated data layers.
• Own Azure Databricks delivery, including notebooks, Workflows/Jobs, Asset Bundles, parameters, concurrency and retries.
• Implement medallion architecture (Landing Staging Curated Analytics) with partitioning, checksums and metadata sidecars.
• Maintain end-to-end pipeline auditability using correlation IDs, trigger records and completion records.
• Partner with SRE teams on data lake/SFTP storage, identity management and secrets for Databricks and related services.
• Implement data quality controls, schema evolution, lineage, cost optimization and runbooks for pipeline incidents.
• Provide curated, quality-assured datasets from data lakes and SQL warehouses to Product and AI teams for RAG and embedding workflows.
• Ensure data contracts cover schema, metadata, access classifications and refresh frequency. Ownership of RAG corpora and vector stores remains with the respective Product and AI teams.

Must-Have Experience

• 10+ years of data engineering experience, including ownership of production data platforms end to end.
• Strong expertise in Azure Databricks, including Spark, Delta Lake, Workflows/Jobs and Asset Bundles.
• Hands-on experience in data lake architecture and SFTP/file-landing ingestion patterns.
• Strong SQL and Python skills.
• Experience with data orchestration and CI/CD for data engineering code.
• Strong understanding of data governance, security isolation, RBAC, auditability and PII handling.
• Experience with investment data and data modelling.

Nice to Have

• Familiarity with RAG and vector-search consumers (building these systems is not required).
• Experience with dbt, Delta Live Tables, Event Hubs or Change Data Capture (CDC).
• Exposure to Databricks SQL and BI serving patterns.

Preferred Domain Experience

Experience in one or more of the following domains is preferred:

• Capital Markets
• Investment Banking
• Asset Management
• Financial Services