Data Engineer

Product-Based Company; Bengaluru; On-site; INR 800,000 - 1,400,000; Full time

Job summary

Product-Based Company in Bengaluru is seeking a data engineer to architect and build scalable data systems across cloud infrastructure. You will own the full lifecycle of data pipelines, from design through performance tuning and maintenance, collaborating with stakeholders to translate business needs into robust data workflows.

The role emphasizes Databricks, Spark, Delta Lake, and Azure Data Factory with Data Lake Storage, requiring strong SQL and Python skills and a track record of optimizing

Qualifications

• Proficient in building and maintaining production data pipelines.
• Strong SQL and Python with PySpark.
• Experience with Databricks as a unified analytics platform.
• Hands-on with Azure Data Factory and Azure Data Lake Storage.
• Understanding Delta Lake ACID transactions.

Responsibilities

• Architect and build scalable data systems across cloud infrastructure.
• Design and optimize ETL workflows ingesting data into enterprise systems.
• Collaborate with stakeholders to translate business needs into data solutions.
• Tune performance of Spark jobs and data pipelines for reliability.
• Partner on architectural decisions for scalable data platforms.

Skills

Databricks

Azure Data Factory

Azure Data Lake Storage

SQL

Python

Spark

Delta Lake

Tools

Apache Spark

Delta Lake

Job description

Batch: 2021 and earlier.

About the role:

• You'll architect and construct data systems that move, transform, and store information at scale across cloud infrastructure
• Your focus spans the full lifecycle of data pipeline development, from initial design through performance tuning and ongoing maintenance
• You'll partner with stakeholders to translate business needs into robust, efficient data workflows

What you'll work on:

• Building and refining ETL workflows that ingest, process, and load data into enterprise systems
• Optimising pipeline performance using Apache Spark and Delta Lake for reliable, fast data operations
• Designing solutions within Azure's ecosystem, leveraging Data Factory for orchestration and Data Lake Storage for distributed data management
• Writing SQL queries and PySpark code to transform raw data into analysis-ready datasets
• Collaborating on architectural decisions to ensure data platforms meet reliability and scalability requirements

What we're looking for:

• Hands-on experience building and maintaining production data pipelines
• Proficiency with Databricks as a unified analytics platform
• Strong command of SQL and Python with PySpark
• Proven track record optimising Apache Spark jobs and data workflows
• Working knowledge of Azure Data Factory and Azure Data Lake Storage
• Understanding of ETL methodologies and Delta Lake for ACID transactions
• Comfort working within cloud-native data platforms and their design patterns