Sr Data Engineer

InfoBeans Technologies Ltd; Indore District; On-site; INR 1,500,000 - 2,100,000; Full time

Job summary

InfoBeans is seeking a Data Engineer in India to design and maintain large-scale data pipelines using Spark/PySpark and SQL. You will optimize jobs, implement deterministic, rule-based data transformations, and design schemas aligned with data volume and access patterns.

The role emphasizes performance tuning and data quality without ML components. The candidate should have 4-6 years of experience in data engineering or distributed data processing and cloud familiarity (Azure/AWS/Google Cloud).

Qualifications

• 4-6 years of experience in Data Engineering or distributed data processing.
• Hands-on experience with Python/PySpark, Apache Spark, Airflow and SQL.
• Familiarity with cloud platforms such as Microsoft Azure (preferable), AWS or Google Cloud.
• Good understanding of schema design, data modeling, and partitioning strategies.
• Practical experience in performance tuning and processing large datasets.
• Strong analytical, problem-solving, and debugging skills.
• Understanding of ETL pipelines, distributed systems, and data quality.

Responsibilities

• Build and maintain scalable data pipelines using Apache Spark/PySpark and SQL.
• Tune and optimize Spark jobs and SQL queries to improve performance and resource utilization.
• Design efficient data schemas based on data volume, access patterns, and business requirements.
• Define appropriate partitioning strategies to improve query performance and processing efficiency.
• Troubleshoot data skew, shuffle overhead, slow queries, and pipeline bottlenecks.
• Implement rule-based data transformations, validations, and business logic.
• Analyze data issues, identify root causes, and validate solutions using metrics and execution plans.

Skills

Python

PySpark

Apache Spark

Airflow

SQL

Azure

AWS

Google Cloud

Schema design

Data modeling

Performance tuning

Data quality

ETL pipelines

Distributed systems

Debugging skills

Tools

Parquet

Iceberg

Delta Lake

Kafka

Scala

Job description

Location Indore CITP / Pune - Baner / Pune - Viman Nagar

Client: Leading Ad-Tech Company / Platform

Experience: 4-6 years

Role Overview

We are looking for a Data Engineer with strong expertise in large-scale data processing, performance optimization, SQL, and data modeling. The role focuses on building efficient data pipelines and implementing deterministic, rule-based business logic. Machine learning is not part of the core requirement.

Required Skills

• 4-6 years of experience in Data Engineering or distributed data processing.
• Strong hands-on experience with Python/PySpark, Apache Spark, Airflow and SQL.
• Familiarity with cloud platforms such as Microsoft Azure (preferable), AWS or Google Cloud.
• Good understanding of schema design, data modeling, and partitioning strategies.
• Practical experience in performance tuning and processing large datasets.
• Strong analytical, problem-solving, and debugging skills.
• Understanding of ETL pipelines, distributed systems, and data quality.

Good to Have

• Experience with Scala, Kafka, Parquet, Iceberg, or Delta Lake.
• Experience in Ad-Tech, advertising analytics, or high-volume event processing.

Key Responsibilities

• Build and maintain scalable data pipelines using Apache Spark/PySpark and SQL.
• Tune and optimize Spark jobs and SQL queries to improve performance and resource utilization.
• Design efficient data schemas based on data volume, access patterns, and business requirements.
• Define appropriate partitioning strategies to improve query performance and processing efficiency.
• Troubleshoot data skew, shuffle overhead, slow queries, and pipeline bottlenecks.
• Implement rule-based data transformations, validations, and business logic.
• Analyze data issues, identify root causes, and validate solutions using metrics and execution plans.

Key Skill Areas

• Performance tuning: Identify bottlenecks and optimize Spark jobs/SQL queries.
• Large datasets: Handle high data volumes efficiently.
• Schema design: Design schemas based on business and query requirements.
• Partitioning: Select suitable partition keys and resolve skew or small-file issues.
• Analytical skills: Debug issues systematically and justify solutions with data.
• Rule-based processing: Implement accurate, deterministic business logic without ML dependencies.