Senior Data Engineer Databricks / PySpark

Wissen Technology; Bengaluru, Mumbai; On-site; INR 1,500,000 - 2,300,000; Full time

Job summary

Wissen Technology is seeking a Senior Data Engineer with strong Databricks, PySpark and Python expertise to design, build and optimize scalable data pipelines. You will handle data ingestion from multiple sources, implement Medallion Architecture and ensure security and governance across banking-scale data systems.

The role requires 6+ years of data engineering experience, production-minded mindset and ability to work across development, QA and production environments.

Qualifications

• Strong hands-on experience with Databricks and PySpark.
• Experience building end-to-end ETL/ELT pipelines.
• Experience with CDC, incremental loading and data quality checks.
• Knowledge of Medallion Architecture (Bronze, Silver, Gold).
• Experience with cloud platforms such as Azure, AWS or OCI.
• Understanding of data security and governance practices.
• Experience in banking/financial services is a plus.

Responsibilities

• Design, develop and maintain scalable end-to-end data pipelines using Databricks and PySpark.
• Build and optimize data ingestion frameworks for large volumes of data.
• Implement Medallion Architecture with Bronze, Silver and Gold layers.
• Develop reusable ingestion frameworks for multiple source systems.
• Integrate data from databases, APIs and cloud storage.
• Implement CDC, schema evolution and data quality checks.
• Handle data quality, deduplication and late-arriving data.
• Optimize Spark jobs and Delta Lake performance.
• Develop production-grade Python code and CI/CD pipelines.
• Ensure data security with RBAC and Unity Catalog.

Tools

Databricks

PySpark

Delta Lake

Python

ETL pipelines

CDC

RBAC

CI/CD

Data security

Git

GitHub

Azure DevOps

Oracle

SAP

Job description

Job Descriptionbelow :Senior Data Engineer Databricks / PySpark

Role: Databricks Developer / Senior Data Engineer

Experience:6+ years in Data Engineering, with strong hands-on experience in Databricks and PySpark

Domain: Banking / Financial Services

Role Overview

We are looking for an experienced Data Engineer with strong expertise in Databricks, PySpark and Python to design, develop and optimize scalable, reliable and secure data pipelines.The ideal candidate should have hands-on experience in processing large datasets, integrating multiple source systems, implementing end-to-end data engineering solutions and working with enterprise-scale data architectures.The candidate should possess strong technical problem-solving skills, experience in production data engineering and the ability to independently design and implement complex data pipelines.

Key Responsibilities

• Design, develop and maintain scalable end-to-end data pipelines using Databricks, PySpark and Python.
• Build and optimize data ingestion frameworks to process large volumes of structured, semi-structured and unstructured data.
• Implement Medallion Architecture comprising Bronze, Silver and Gold layers.
• Develop metadata-driven and reusable ingestion frameworks to support multiple source systems.
• Integrate data from databases, REST APIs, cloud storage, Oracle, SAP and various file formats such as CSV, JSON and Parquet.
• Implement incremental data loading, Change Data Capture (CDC), schema evolution and data quality validation.
• Handle duplicate records, late-arriving data, schema mismatches and data drift.
• Optimize Spark jobs using partitioning, Adaptive Query Execution (AQE), broadcast joins and Delta Lake optimization techniques.
• Implement robust error handling, retry mechanisms, logging, monitoring and production troubleshooting.
• Develop modular, reusable and maintainable Python code for data engineering applications.
• Implement CI/CD pipelines and manage code deployment across development, QA and production environments.
• Ensure data security through access control, encryption, data masking and role-based security mechanisms.
• Collaborate with business stakeholders, architects and technical teams to understand data requirements and deliver reliable data solutions.

Required Technical SkillsDatabricks & Apache Spark

• Strong hands-on experience with Databricks and PySpark.
• Good understanding of Spark architecture, Driver, Executors and memory management.
• Experience with Delta Lake, Delta tables and Medallion Architecture.
• Knowledge of Spark optimization, AQE, partitioning and performance tuning.
• Experience with Databricks Workflows and orchestration.

Python Programming

• Strong proficiency in Python for data engineering applications.
• Experience developing modular, reusable and production-grade Python code.
• Understanding of multithreading, multiprocessing and asynchronous execution.
• Experience with exception handling, logging and testing frameworks.

Data Engineering & ETL

• Experience building end-to-end ETL/ELT pipelines.
• Strong understanding of metadata-driven ingestion frameworks.
• Experience with CDC, incremental loading and data quality checks.
• Knowledge of schema evolution, deduplication and late-arriving data.
• Experience handling large datasets and integrating multiple source systems.

Cloud & Data Integration

• Experience with cloud platforms such as Azure, AWS or OCI.
• Knowledge of cloud storage solutions such as Azure Blob Storage, ADLS or Amazon S3.
• Experience integrating databases, APIs and enterprise source systems.
• Exposure to Oracle, SAP or banking data systems is desirable.

DevOps & Production Engineering

• Experience with Git, GitHub or Azure DevOps.
• Knowledge of CI/CD pipelines and branching strategies.
• Experience with unit testing, integration testing and production deployments.
• Ability to troubleshoot production failures and perform root cause analysis.

Security & Governance

• Understanding of role-based access control (RBAC).
• Experience with Databricks Unity Catalog and access management.
• Knowledge of PII data protection, encryption and data masking.
• Understanding of data governance and enterprise security practices.

Preferred Qualifications

• Experience in banking, financial services or other regulated industries.
• Experience handling high-volume transactional datasets.
• Exposure to Oracle Fusion, SAP, OCI or enterprise data integration.
• Experience implementing data security and governance frameworks.
• Exposure to AI-assisted development tools such as GitHub Copilot.

Candidate Expectations

The candidate should demonstrate:

• Strong hands-on technical expertise.
• Experience working on complex, enterprise-scale data engineering projects.
• Ability to independently design and implement scalable data pipelines.
• Strong analytical and problem-solving skills.
• Ability to explain technical architecture and implementation decisions.
• Ownership of development, deployment and production support activities.