Senior Data Engineer

People, Jobs, and News; Nagpur District; On-site; INR 1,400,000 - 2,100,000; Full time

Job summary

People, Jobs, and News is seeking a Senior Data Engineer to design, build, and optimize our Ingest Factory and Data Processing Frameworks using Python, PySpark, and the Databricks Lakehouse ecosystem. You will implement metadata-driven pipelines and reusable data frameworks with strong software engineering discipline.

Collaborating in an Agile environment, you will own testing, governance, and orchestration, while tuning performance of large Spark workloads and ensuring scalable, maintainable

Qualifications

• 5+ years of production cloud data engineering experience.
• Deep hands-on Databricks notebooks, jobs optimization, and Delta Lake usage.
• Advanced Python and Spark with distributed workloads understanding.
• 3–5+ years applying SOLID coding principles and version control.
• Production experience with Terraform and CI/CD pipelines.
• Proficient in writing and optimizing complex SQL queries.
• Emphasis on data quality, governance, and scalable architectures.

Responsibilities

• Ingest Factory Design: build robust data ingestion frameworks using LakeFlow and connectors.
• Data Lakehouse Patterns: implement ETL/ELT patterns with Delta Lake and governance.
• Metadata-driven orchestration: create parameterized notebooks and end-to-end flows.
• Advanced Python development: clean, modular code and reusable internal packages.
• Framework creation: define abstractions to boost team development efficiency.
• Testing & QA: implement unit, integration, and end-to-end tests for data pipelines.
• Performance tuning: optimize Spark workloads and SQL queries to reduce costs.
• DevOps & collaboration: manage CI/CD and IaC; participate in SCRUM and agile delivery.

Skills

Cloud data engineering

Databricks ecosystem

Advanced Python & Spark

SOLID coding principles

CI/CD & DevOps

Terraform / IaC

SQL optimization

Data quality & governance

Metadata-driven pipelines

Tools

Databricks

Delta Lake

LakeFlow

Terraform

GitHub Actions

GitLab CI

Job description

Job Description: Senior Data Engineer – Data Ingestion & Platforms

Role Overview

We are seeking a seasoned Senior Data Engineer with a strong software engineering mindset to design,

build, and optimize our next-generation Ingest Factory and Data Processing Frameworks. In this role,

you will go beyond traditional ETL scripting to build scalable, metadata-driven pipelines and reusable

data frameworks.

The ideal candidate possesses deep expertise in Python, PySpark, and the Databricks Lakehouse

ecosystem (including LakeFlow and Delta Lake), combined with rigorous software engineering

discipline (SOLID, CI/CD, and infrastructure as code). You will work both independently and

collaboratively within an Agile environment to build production-grade software that ensures data quality,

governance, and seamless orchestration.

Key Responsibilities

Architecture & Pipeline Engineering

• Ingest Factory Design: Design, develop, and maintain robust data ingestion frameworks

leveraging Databricks LakeFlow, managed connectors, and declarative pipelines.

• Data Lakehouse Patterns: Implement repeatable ETL/ELT patterns within a Delta Lake

architecture, ensuring optimized storage, table design, and strict data lineage enforcement.

• Metadata-Driven Orchestration: Build parameterized notebooks and end-to-end orchestration

flows to automate ingestion across diverse source system patterns.

Software Craftsmanship & Automation

• Advanced Python Development: Write clean, modular, and maintainable Python code applying

SOLID and DRY principles. Move beyond basic PySpark scripting to contribute to and publish

reusable internal packages (e.g., PyPI).

• Framework Creation: Define reusable functions and framework-level abstractions to

dramatically improve development efficiency across the data team.

• Testing & Quality Assurance: Implement rigorous data quality checks, monitoring, and alerting

frameworks. Lead test practices including Unit, Integration, and End-to-End (E2E) testing.

Optimization & Troubleshooting

• Performance Tuning: Optimize complex distributed Spark workloads, Databricks compute

configurations, and SQL queries (efficient filtering, indexing, and joins) to reduce processing

costs.

• Advanced Troubleshooting: Deep dive into Spark UI and logs to diagnose and resolve

performance bottlenecks, data skew, and serialization issues.

DevOps & Collaboration

• CI/CD & IaC: Own the deployment lifecycle by building and maintaining GitHub Actions / GitLab

pipelines and provision infrastructure utilizing Terraform (IaC).

• Agile Delivery: Actively participate in SCRUM ceremonies, design solutions to specific user

stories, vet architectures with the team, and deliver retro demos prior to production deployment.

Technical Skillset & Qualifications

Must-Have Core Skills:

• Cloud Data Experience: 5+ years of production experience in cloud data engineering and building

enterprise-grade software.

• Databricks Ecosystem: Deep hands-on experience with Databricks Notebooks, Jobs

optimization, Delta Lake, Connectors, and LakeFlow (jobs, tasks, flows).

• Advanced Python & Spark: Mastery of Core Python, Data Structures & Algorithms (DSA), and

package management. Clear understanding of distributed workloads (Spark vs. single-node

processing).

• Software Engineering Disciplines: 3–5+ years of practicing SOLID coding principles, Git controls

(PR reviews, branching strategies), and modern IDE features (Cursor/VSCode).

• DevOps & Automation: Production experience with Terraform and CI/CD tools (GitHub Actions

or GitLab CI).

• Advanced SQL: Proficiency in writing and optimizing mid-to-complex queries, ensuring efficient

data processing and model design.

Soft Skills & Operational Traits

• Ability to work independently as a self-starter while being a highly collaborative team player.
• Strong business and data literacy—understanding not just how to move data, but the business

purpose behind it.

• Excellent communication skills for vetting solutions with peer engineers and presenting work

during retro demos.

Skills: python,data engineer,pyspark,spark,metadata,metadata driven pipeline,deployment,customization,configuration,data packages,data products