Data Engineer

Sourcebae; Bengaluru; On-site; INR 2,500,000 - 4,000,000; Full time

Job summary

Sourcebae is seeking a Senior Data Engineer to design and build reliable batch and streaming data platforms in Bengaluru. The role requires advanced Python and SQL skills, mandatory hands-on Apache Spark experience, and strong data modelling with practical cloud platform expertise.

You will own ingestion, transformation, quality, orchestration, performance and operational reliability across data pipelines, enabling trusted data products for analytics and ML teams.

Qualifications

• 5+ years of relevant experience in data engineering or cloud data platforms.
• Strong Python and advanced SQL skills.
• Hands-on Apache Spark experience including performance tuning.

Responsibilities

• Design and build scalable ETL/ELT pipelines for structured, semi-structured and streaming data.
• Develop distributed data-processing workloads using Apache Spark and Python/PySpark.
• Implement ingestion, transformation, validation, reconciliation and publishing workflows.
• Design cloud data lake, lakehouse and data-warehouse components.
• Implement incremental loading, change data capture and idempotent processing.
• Optimise Spark jobs, SQL queries, storage formats and cloud resource utilisation.
• Build data-quality checks, monitoring, alerting and recovery mechanisms.
• Collaborate with analytics, AI/ML and application teams to deliver trusted data products.

Skills

Python

SQL

Data modelling

Orchestration

Batch & streaming

Tools

Apache Spark

Job description

We are seeking a Senior Data Engineer to design and build reliable batch and streaming data platforms. The role requires advanced Python and SQL, mandatory hands-on Apache Spark experience, strong data modelling and practical expertise with at least one cloud platform. The engineer will own ingestion, transformation, quality, orchestration, performance and operational reliability.

Key responsibilities

• Design and build scalable ETL/ELT pipelines for structured, semi-structured and streaming data.
• Develop distributed data-processing workloads using Apache Spark and Python/PySpark.
• Implement ingestion, transformation, validation, reconciliation and publishing workflows.
• Design cloud data lake, lakehouse and data-warehouse components.
• Implement incremental loading, change data capture, schema evolution and idempotent processing.
• Optimise Spark jobs, SQL queries, storage formats, partitioning and cloud resource utilisation.
• Build data-quality checks, monitoring, alerting, retry and recovery mechanisms.
• Implement orchestration and dependency management for production data pipelines.
• Collaborate with analytics, AI/ML and application teams to deliver trusted data products.
• Maintain documentation covering source-to-target mapping, lineage, operational runbooks and architecture.

Required qualifications and experience

• 5+ years of relevant experience in data engineering, big-data engineering or cloud data platforms.
• Strong Python and advanced SQL skills.
• Mandatory hands-on experience with Apache Spark, including performance tuning and distributed-processing concepts.
• Experience with at least one cloud platform: AWS, Microsoft Azure or Google Cloud Platform.
• Experience building and supporting production-grade data pipelines.
• Strong understanding of data modelling, data quality, orchestration and operational reliability.