Senior Data Engineer

Jobtailor; Mumbai; On-site; INR 1,500,000 - 2,100,000; Full time

Job summary

Jobtailor is seeking an experienced Data Engineer to design, develop, and maintain scalable data solutions using Databricks, Spark (PySpark), SQL, and Delta Lake. You will lead data engineering activities and collaborate with stakeholders to deliver enterprise-grade data platforms.

Responsibilities include building robust ingestion pipelines, automated testing frameworks, and CI/CD automation using Azure DevOps or GitHub, with a strong focus on data quality, governance, and security.

Qualifications

• 5+ years of experience in Data Engineering or related fields.
• Strong hands-on expertise in Databricks, Spark (PySpark), SQL, and Delta Lake.
• Experience with Data Warehousing, Lakehouse architectures and data integration.
• Familiarity with CI/CD pipelines using Azure DevOps or GitHub.
• Knowledge of data quality, governance, and metadata practices.

Responsibilities

• Design, develop, and maintain scalable data solutions across platforms.
• Lead data engineering activities from requirements to deployment and support.
• Build and optimize ingestion, transformation, and orchestration pipelines.
• Develop automated testing, validation, and QA frameworks for data platforms.
• Collaborate with stakeholders to deliver enterprise-grade data solutions.
• Implement security controls and data governance standards across environments.

Skills

Databricks

Apache Spark (PySpark)

SQL

Data Engineering

CI/CD Pipelines

Stakeholder Management

Leadership

Problem-Solving Skills

Education

Bachelor's Degree

Tools

Microsoft Fabric

Azure DevOps

GitHub

Power BI

Terraform

Bicep

ARM Templates

Delta Lake

Job description

• Design, develop, and maintain scalable data solutions using Databricks, Microsoft Fabric, SQL, Spark, and Delta Lake


• Build and optimize data ingestion, transformation, and orchestration pipelines for enterprise data platforms


• Develop and implement automation frameworks for data validation, testing, monitoring, deployment, and operational support


• Lead data engineering activities across requirements, design, development, testing, deployment, and support


• Build and maintain automated QA and reconciliation frameworks for analytical and reporting solutions


• Validate data quality, completeness, consistency, and business rule compliance across enterprise datasets


• Develop and maintain semantic models, lakehouses, warehouses, and data marts within Microsoft Fabric


• Implement and validate security controls, including Row-Level Security (RLS) and data governance standards


• Collaborate with business stakeholders, architects, analytics teams, and platform engineers to deliver enterprise-grade solutions


• Support CI/CD, DevOps, Infrastructure-as-Code, and deployment automation initiatives


• Drive performance optimization and cost efficiency across Databricks and Fabric environments


• Establish engineering best practices, coding standards, reusable frameworks, and governance processes

Requirements

• Bachelor's Degree


• 5+ years of experience in Data Engineering, Analytics Engineering, or Data Platform Development


• Strong hands-on expertise in Databricks, Apache Spark (PySpark), SQL, Microsoft Fabric, Delta Lake, and Data Warehousing


• Experience designing and implementing Data Pipelines, ETL/ELT Solutions, Lakehouse Architectures, and Data Integration Frameworks


• Strong experience with Databricks Workflows, Jobs, Automation, and Orchestration


• Experience developing automated testing and validation frameworks for data platforms


• Hands-on experience implementing CI/CD pipelines using Azure DevOps, GitHub, or equivalent tools


• Strong understanding of data quality, reconciliation, metadata management, and governance practices


• Experience with performance tuning and optimization of Spark workloads


• Experience with Microsoft Fabric components, including Data Factory, Lakehouse, Warehouse, Semantic Models, and Power BI


• Preferred: experience with Infrastructure as Code (Terraform, Bicep, ARM Templates), DataOps, MLOps, Power BI semantic models, DAX, analytical reporting, enterprise data governance and security frameworks, cloud platforms such as Azure, AWS, or GCP, and technical leadership/mentoring


• Strong analytical and problem-solving skills


• Excellent communication and stakeholder management abilities


• Ability to lead initiatives independently and work effectively in cross-functional teams


• Strategic mindset with the ability to align technical solutions with business objectives

Core Competencies

Demonstrates expertise in designing and developing scalable data solutions using Databricks, SQL, and Spark, while ensuring data quality and governance. Proficient in implementing CI/CD pipelines and automation frameworks to optimize data engineering processes and support enterprise data platforms.

Highest-signal resume keywords

• Databricks


• SQL


• Apache Spark (PySpark)


• Data Engineering


• CI/CD Pipelines

Hard Skills

• Data Pipeline Design


• ETL/ELT Solutions


• Lakehouse Architecture


• Data Integration Frameworks


• Automated Testing Frameworks


• Performance Tuning


• Data Quality Management


• Data Warehousing


• Data Governance


• Automation Frameworks

Soft Skills

• Analytical Skills


• Problem-Solving Skills


• Communication Skills


• Stakeholder Management


• Leadership

Certifications & Qualifications

• Bachelor's Degree

Industry Keywords

• Data Engineering


• Data Analytics


• Data Governance


• Data Quality


• Cloud Platforms

Tools & Technologies

• Microsoft Fabric


• Delta Lake


• Azure DevOps


• GitHub


• Power BI


• Terraform


• Bicep


• ARM Templates