Senior Data Engineer

TechWise Digital; Bengaluru; On-site; INR 1,800,000 - 3,200,000; Full time

Job summary

TechWise Digital in Bengaluru seeks a Senior Data Engineer to design and build scalable batch and streaming data platforms. You will develop production-grade ETL/ELT pipelines using Python, SQL, Spark, and AWS to ingest, transform, validate, reconcile and publish data workflows.

The role focuses on designing data lakes, lakehouses and warehouse components, enabling incremental loading, CDC, schema evolution and idempotent processing, while optimizing Spark jobs and cloud resources, and ensuring

Qualifications

• 5+ years of relevant Data Engineering experience.
• Strong Python and advanced SQL skills.
• Mandatory hands-on Apache Spark experience including performance tuning.
• Strong knowledge of data modelling, data quality, orchestration and production data pipelines.
• Experience with AWS cloud platforms is essential.

Responsibilities

• Design cloud data lakes, lakehouses and data warehouse components.
• Implement incremental loading, CDC, schema evolution and idempotent processing.
• Optimize Spark jobs, SQL queries, partitioning and cloud resources.
• Implement data-quality checks, monitoring, alerting, orchestration, recovery mechanisms and operational processes.

Skills

Python

SQL

Apache Spark/PySpark

AWS

Job description

We are looking for a Senior Data Engineer to design and build scalable, reliable batch and streaming data platforms. The role involves developing production-grade ETL/ELT pipelines, data ingestion, transformation, validation, reconciliation, and publishing workflows. The candidate will work extensively with Python, SQL, Apache Spark/PySpark, and AWS to process large-scale data.

Key responsibilities include designing cloud data lakes, lakehouses and data warehouse components; implementing incremental loading, CDC, schema evolution and idempotent processing; and optimizing Spark jobs, SQL queries, partitioning and cloud resources. The role also involves implementing data-quality checks, monitoring, alerting, orchestration, recovery mechanisms and operational processes.

Requirements

5+ years of relevant Data Engineering experience, strong Python and advanced SQL skills, and mandatory hands-on Apache Spark experience including performance tuning. Strong knowledge of data modelling, data quality, orchestration and production data pipelines is required. Experience with AWS cloud platforms is essential.

Skills: apache spark,python,aws