Senior Data Engineer

People, Jobs, and News; Dadri; On-site; INR 1,500,000 - 2,500,000; Full time

Job summary

People, Jobs, and News is seeking a Senior Data Engineer to focus on designing, building, and optimizing data ingestion frameworks. The role requires expertise in Python, PySpark, and the Databricks ecosystem, ensuring scalable and metadata-driven pipelines.

You will work on performance tuning, testing, and CI/CD processes while actively engaging in Agile methodologies. Strong communication skills and a collaborative attitude are essential for success in this role.

Qualifications

• 5+ years of production experience in cloud data engineering and building enterprise-grade software.
• Deep hands-on experience with Databricks Notebooks, Jobs optimization, Delta Lake, and LakeFlow.
• Proficiency in Core Python, Data Structures & Algorithms, and understanding of distributed workloads.

Responsibilities

• Design, develop, and maintain data ingestion frameworks using Databricks.
• Optimize distributed Spark workloads and SQL queries to improve performance.
• Lead testing practices including Unit, Integration, and End-to-End testing.

Skills

Python

PySpark

Databricks Lakehouse

SQL

Data Structures & Algorithms

Terraform

CI/CD

Agile methodologies

Job description

Overview

Senior Data Engineer – Data Ingestion & Platforms. Role focuses on designing, building, and optimizing next-generation Ingest Factory and Data Processing Frameworks to deliver scalable, metadata-driven pipelines and reusable data frameworks. Requires deep expertise in Python, PySpark, and the Databricks Lakehouse ecosystem (including LakeFlow and Delta Lake), coupled with strong software engineering discipline (SOLID, CI/CD, and infrastructure as code). Works independently and in Agile teams to deliver production-grade software that ensures data quality, governance, and seamless orchestration.

Responsibilities

• Ingest Factory Design: Design, develop, and maintain robust data ingestion frameworks leveraging Databricks LakeFlow, managed connectors, and declarative pipelines.
• Data Lakehouse Patterns: Implement repeatable ETL/ELT patterns within a Delta Lake architecture, ensuring optimized storage, table design, and strict data lineage enforcement.
• Metadata-Driven Orchestration: Build parameterized notebooks and end-to-end orchestration flows to automate ingestion across diverse source system patterns.
• Advanced Python Development: Write clean, modular Python code applying SOLID and DRY principles; contribute to and publish reusable internal packages (e.g., PyPI).
• Framework Creation: Define reusable functions and framework-level abstractions to dramatically improve development efficiency across the data team.
• Testing & Quality Assurance: Implement rigorous data quality checks, monitoring, and alerting frameworks; lead testing practices including Unit, Integration, and End-to-End (E2E) testing.
• Performance Tuning: Optimize complex distributed Spark workloads, Databricks compute configurations, and SQL queries to reduce processing costs.
• Advanced Troubleshooting: Deep dive into Spark UI and logs to diagnose and resolve performance bottlenecks, data skew, and serialization issues.
• CI/CD & IaC: Own the deployment lifecycle by building and maintaining GitHub Actions / GitLab pipelines and provisioning infrastructure using Terraform (IaC).
• Agile Delivery: Actively participate in SCRUM ceremonies, design solutions to user stories, vet architectures with the team, and deliver retro demos prior to production deployment.

Technical Skillset & Qualifications

• Must-Have Core Skills:
• Cloud Data Experience: 5+ years of production experience in cloud data engineering and building enterprise-grade software.
• Databricks Ecosystem: Deep hands-on experience with Databricks Notebooks, Jobs optimization, Delta Lake, Connectors, and LakeFlow (jobs, tasks, flows).
• Advanced Python & Spark: Proficiency in Core Python, Data Structures & Algorithms (DSA), and package management; clear understanding of distributed workloads (Spark vs. single-node processing).
• Software Engineering Disciplines: 3–5+ years of practicing SOLID coding principles, Git controls (PR reviews, branching strategies), and modern IDE features.
• DevOps & Automation: Production experience with Terraform and CI/CD tools (GitHub Actions or GitLab CI).
• Advanced SQL: Proficiency in writing and optimizing mid-to-complex queries, ensuring efficient data processing and model design.
• Soft Skills & Operational Traits:
• Ability to work independently as a self-starter while being a highly collaborative team player.
• Strong business and data literacy—understanding not just how to move data, but the business purpose behind it.
• Excellent communication skills for vetting solutions with peer engineers and presenting work during retro demos.
• Additional Keywords: data ingestion & platforms, data engineer, pyspark, python, sql, data structure & algorithms.