Senior Manager

Pyramid It Consulting; Hyderabad, Chennai District, Bengaluru; Hybrid; INR 3,000,000 - 7,000,000; Full time

Job summary

Pyramid It Consulting is seeking an experienced Databricks Data Architect / Lead Data Engineer for BFSI & Banking. You will design modern Data Warehouse/Data Lakehouse solutions using Databricks on cloud platforms, build scalable data pipelines, and ensure data quality and governance.

Ideal candidates have 13+ years in Data & Analytics, hands-on Databricks, SQL, Python, PySpark, and strong BFSI domain experience.

Qualifications

• Experience in designing/implementing modern Data Warehouse and Lakehouse architectures.
• Strong hands-on with Databricks, SQL, Python, PySpark, and cloud platforms (AWS/Azure).
• BFSI/Banking domain project experience and regulatory reporting insights are essential.
• Proven leadership, solution design, client-facing communications, and mentoring.
• Ability to collaborate with multiple tech and business stakeholders to deliver data pipelines.

Responsibilities

• Design and develop modern Data Warehouse and Data Lakehouse solutions using Databricks on AWS/Azure.
• Define scalable, high-performance data architectures aligned with analytics needs.
• Provide innovative data engineering and analytics solutions across platforms.
• Collaborate with Data Warehouse, BI, Business, and Tech leads to scope ETL/data pipelines.
• Develop and maintain robust ETL/ELT pipelines for batch and streaming data.
• Transform data using SQL, Python, PySpark, Spark, and Databricks.
• Build Delta Lake/Lakehouse architectures including ingestion, storage and consumption layers.
• Translate reporting requirements into data models and reporting layers.
• Triage production issues, perform root cause analysis, and implement fixes.
• Monitor data pipelines for quality, availability, and performance.
• Orchestrate pipelines with Apache Airflow; integrate with cloud and Databricks platforms.
• Implement batch and streaming data tech (Kafka, AWS Kinesis) where applicable.
• Lead technical discussions, design reviews, and architectural decisions.
• Mentor junior team members and promote engineering best practices.

Skills

Data Warehousing

ETL/ELT

Databricks

Python

SQL

PySpark

Cloud (AWS/Azure)

BFSI domain experience

Data Modeling

Leadership

Stakeholder mgmt

Education

Bachelor's/Master's in CS/IT/Engineering/Data Science

Tools

AWS

Azure

SQL

Python

Airflow

Delta Lake

Git

Terraform

CircleCI

Kinesis

Kafka

Job description

Job Description Databricks Data Architect / Lead Data Engineer BFSI & Banking

Position Overview

We are looking for an experienced Databricks Data Architect / Lead Data Engineer with strong expertise in Data Warehousing, ETL, Databricks, Cloud Data Platforms, and Data Engineering. The ideal candidate should have extensive experience delivering large-scale data and analytics solutions, with strong hands‑on expertise in Databricks, SQL, Python, PySpark, AWS/Azure, and modern data architecture.

The candidate must have strong experience in the BFSI (Banking, Financial Services & Insurance) domain, preferably with hands‑on experience working on Banking projects, financial data platforms, regulatory reporting, customer/account data, transactions, risk, compliance, or related banking data solutions.

Roles and Responsibilities

• Design and develop modern Data Warehouse and Data Lakehouse solutions using Databricks and cloud platforms such as AWS and Azure.
• Define and implement scalable, high‑performance data architecture and data engineering solutions aligned with business and analytical requirements.
• Provide forward‑thinking and innovative solutions in the Data Engineering, Data Analytics, and Data Platform space.
• Collaborate with Data Warehouse, BI, Business, and Technical Leads to understand requirements for new ETL/data pipeline development.
• Design, develop, and maintain robust ETL/ELT pipelines for batch and near‑real‑time/streaming data processing.
• Develop data transformation processes using SQL, Python, PySpark, Spark, and Databricks.
• Build and optimize Delta Lake/Lakehouse architectures, including ingestion, transformation, storage, and consumption layers.
  Work closely with business stakeholders to understand reporting requirements and translate them into effective data models and reporting‑layer solutions.
• Analyze and triage production issues, identify gaps in existing pipelines, perform root‑cause analysis, and implement permanent fixes.
• Monitor and troubleshoot ETL/data pipelines, ensuring data quality, availability, reliability, and performance.
• Orchestrate data pipelines using Apache Airflow and integrate workflows with cloud and Databricks platforms.
• Implement data engineering solutions using batch and streaming technologies, including Kafka and AWS Kinesis, where applicable.
• Drive technical discussions with client architects, business stakeholders, engineering teams, and other technical leads.
• Participate in solution architecture, technical design, code reviews, performance optimization, and implementation discussions.
• Mentor and support junior/entry‑level team members in resolving technical challenges and production issues.
• Provide technical guidance and help establish engineering best practices, coding standards, and reusable frameworks.
• Work within a DevOps/CI‑CD environment using tools such as Git, Terraform, CircleCI, and related technologies.
  Implement and support data governance, data management, security, metadata, and data quality practices.
• Work with modern Databricks capabilities including Unity Catalog, Delta Lake, data sharing, and related Data & AI platform capabilities.
  Collaborate with cross‑functional Agile teams and actively participate in Scrum ceremonies, sprint planning, estimation, technical discussions, and retrospectives.
  Support performance tuning of complex SQL queries, Spark jobs, ETL pipelines, and data warehouse processes.
  Ensure solutions adhere to enterprise architecture, security, compliance, and data governance standards, particularly within the BFSI/Banking environment.
  Work with modern Databricks capabilities including Unity Catalog, Delta Lake, data sharing, and related Data & AI platform capabilities.
• Collaborate with cross‑functional Agile teams and actively participate in Scrum ceremonies, sprint planning, estimation, technical discussions, and retrospectives.
• Support performance tuning of complex SQL queries, Spark jobs, ETL pipelines, and data warehouse processes.
• Ensure solutions adhere to enterprise architecture, security, compliance, and data governance standards, particularly within the BFSI/Banking environment.
  Experience
• 13+ years of overall experience in Data & Analytics, with strong experience in Data Engineering/Data Warehousing.
• Experience delivering at least 2 large‑scale, end‑to‑end Data Warehouse/Data Engineering implementations.
• Strong experience in BFSI/Banking domain projects is mandatory.
• Proven experience in technical leadership, solution design, architecture, and client‑facing technical discussions.
• Strong communication, stakeholder management, presentation, and technical leadership skills.

Education

• Bachelor's and/or Master's degree in Computer Science, Information Technology, Engineering, Data Science, or a related field.
• Equivalent professional experience may be considered.Role & responsibilities.