Senior Data Engineer

Jobtailor; Bengaluru; On-site; INR 1,800,000 - 3,000,000; Full time

Job summary

Jobtailor in Bengaluru is seeking an experienced data engineer to design, build, and scale batch and streaming pipelines using PySpark, Azure Databricks, and Microsoft Fabric. You will implement the Medallion Architecture across Bronze, Silver, and Gold layers to support enterprise analytics and data products.

You will optimize Spark workloads, ensure data quality, governance, and lineage, and collaborate with architects and product teams in an Agile environment.

Qualifications

• 6+ years of proven Data Engineering experience in enterprise cloud platforms.
• Hands-on expertise with PySpark, Spark SQL, DataFrame APIs, debugging and performance optimization.
• Strong experience with Azure Databricks, notebooks, jobs/workflows, clusters, Delta Lake and production deployment patterns.
• Practical knowledge of Medallion Architecture (Bronze/Silver/Gold) and data lakehouse concepts.

Responsibilities

• Design, develop, and maintain scalable batch and streaming data pipelines using PySpark, Azure Databricks, and Fabric.
• Architect and implement Medallion Architecture across Bronze, Silver, and Gold layers for enterprise data processing and analytics.
• Design secure, governed data-sharing solutions and data products across workspaces and external consumers.

Skills

PySpark

Spark SQL

Data Modeling

ETL/ELT

Data Quality

Performance Tuning

SQL Proficiency

Stakeholder Mgmt

Education

Bachelor's degree in CS/Engineering/IS

Tools

Azure Databricks

Microsoft Fabric

Delta Lake

Azure Data Factory

Azure Data Lake Storage Gen2

OneLake

Git/CI-CD

Azure Synapse Analytics

Job description

• Design, develop, and maintain scalable batch and streaming data pipelines using PySpark, Azure Databricks, and Microsoft Fabric
• Architect and implement Medallion Architecture across Bronze, Silver, and Gold layers for enterprise data processing and analytics
• Build robust ETL/ELT solutions for data ingestion, transformation, validation, reconciliation, and delivery across multiple source systems
• Develop and optimize PySpark and Spark SQL workloads for high-volume structured, semi-structured, and unstructured data
• Design and maintain Lakehouse and data lake solutions using Azure Data Lake Storage Gen2, Delta Lake, Microsoft Fabric OneLake, Fabric Lakehouse, and Warehouse
• Implement integration solutions using Azure Data Factory, Fabric Data Factory, data pipelines, notebooks, and Dataflows Gen2
• Design secure and governed data-sharing solutions across workspaces, domains, business units, and approved external consumers
• Implement reusable data products and cross-workspace sharing patterns using OneLake, OneLake shortcuts, Lakehouse, Warehouse, and semantic models
• Contribute to Microsoft Fabric capacity planning, workspace-to-capacity assignment, workload monitoring, utilization analysis, and performance optimization
• Monitor Fabric workloads using available capacity and workload metrics, identify resource contention, and recommend workload or scheduling improvements
• Design domain-aligned Fabric workspace structures with appropriate separation for development, testing, production, security, and ownership boundaries
• Implement data quality controls, monitoring, observability, lineage, error handling, reconciliation, and auditability across data pipelines
• Apply security best practices using managed identities, role-based access control, workspace roles, row-level or object-level controls where applicable, and secure secrets management
• Integrate data engineering solutions with Git-based source control and CI/CD pipelines for automated testing and deployment
• Optimize performance, scalability, reliability, and cost across Azure Databricks and Microsoft Fabric workloads
• Collaborate with Data Architects, Product Owners, Business Analysts, Data Scientists, QA engineers, governance teams, and platform teams in an Agile/Scrum environment
• Provide technical leadership, conduct design and code reviews, establish engineering standards, and mentor data engineers

Requirements

• Bachelor's degree in Computer Science, Engineering, Information Systems, or a related field
• 6+ years of proven Data Engineering experience, including delivery of enterprise-scale cloud data platforms
• Excellent hands-on expertise in PySpark, Spark SQL, DataFrame APIs, debugging, reusable framework development, and performance optimization
• Strong hands-on experience with Azure Databricks, including notebooks, jobs/workflows, clusters, Delta Lake, and production deployment patterns
• Practical implementation experience with Medallion Architecture, including Bronze, Silver, and Gold data layers
• Strong knowledge of ETL/ELT processes, ingestion patterns, incremental processing, transformation frameworks, and workflow orchestration
• Hands-on Microsoft Fabric experience, including Fabric Data Factory, notebooks, Lakehouse, Warehouse, OneLake, data pipelines, Dataflows Gen2, and semantic models
• Experience with Fabric capacity concepts, capacity assignment, utilization monitoring, workload analysis, performance tuning, and capacity-aware solution design
• Experience designing Fabric workspaces across domains and environments, including access, ownership, deployment, and workload-isolation considerations
• Strong experience implementing enterprise data-sharing patterns using OneLake shortcuts, shared Lakehouse or Warehouse data, governed data products, and semantic models
• Hands-on Azure experience, including Azure Data Lake Storage Gen2, Azure Data Factory, Azure Synapse Analytics, Azure Key Vault, and Azure Monitor
• Advanced SQL skills with experience in data modelling, data warehousing, query optimization, and performance tuning
• Experience with Delta Lake, Parquet, schema evolution, partitioning, and modern Lakehouse architecture patterns
• Strong understanding of data quality, metadata management, governance, lineage, security, privacy, and operational monitoring
• Experience with Azure DevOps or equivalent Git-based repositories and CI/CD pipelines for data engineering solutions
• Excellent problem-solving, troubleshooting, communication, and stakeholder-management skills
• Preferred: Experience delivering Microsoft Fabric solutions in enterprise environments with multiple workspaces and shared capacity
• Preferred: Experience interpreting capacity and workload metrics and recommending performance, scheduling, scaling, or workload-distribution improvements
• Preferred: Knowledge of streaming architectures, event-driven processing, Eventstreams, and real-time analytics
• Azure Data Engineer, Azure Databricks, or Microsoft Fabric certifications are preferred

Core Competencies

Demonstrates expertise in designing and implementing scalable data pipelines and architectures using PySpark, Azure Databricks, and Microsoft Fabric. Proficient in ETL/ELT processes, data quality management, and performance optimization across cloud data platforms.

Highest-signal resume keywords

• PySpark Development
• Azure Databricks
• Medallion Architecture
• ETL/ELT Processes
• Data Quality Management

ATS Optimization Keywords

Hard Skills

• Data Engineering
• Spark SQL
• DataFrame APIs
• Performance Optimization
• Data Ingestion
• Data Transformation
• Data Warehousing
• SQL
• Data Modeling
• Capacity Planning

Soft Skills

• Problem-Solving
• Communication
• Stakeholder Management
• Collaboration
• Technical Leadership

Certifications & Qualifications

• Azure Data Engineer
• Azure Databricks Certification
• Microsoft Fabric Certification

Industry Keywords

• Cloud Data Platforms
• Data Pipelines
• Data Governance
• Data Quality Controls
• Agile/Scrum Environment

Tools & Technologies

• Azure Data Lake Storage Gen2
• Azure Data Factory
• Microsoft Fabric
• Delta Lake
• OneLake
• Dataflows Gen2
• Azure Synapse Analytics
• Azure Key Vault
• Azure Monitor
• Git