Data Engineer

Intact Green Services (india); Bengaluru; On-site; INR 1,800,000 - 2,800,000; Full time

Benefits offered by this job

Industry-standard compensation

Job summary

Intact Green Services (india) is seeking a skilled PySpark Data Engineer in Bangalore to design and build scalable data pipelines, optimize distributed data processing, and implement efficient ETL workflows using PySpark and cloud-native technologies.

You will collaborate with data scientists, analysts, and engineering teams to ensure reliable data ingestion, transformation, and delivery across sophisticated data platforms and applications.

Qualifications

• Strong hands-on experience in PySpark development.
• Advanced programming in Python with NumPy and Pandas.
• Expertise in writing advanced SQL queries.
• Experience building complex ETL pipelines and data transformations.
• Ability to process large datasets and nested JSON structures.
• Experience with distributed data processing systems.

Responsibilities

• Design, develop, and maintain scalable PySpark data pipelines.
• Fetch and ingest data from multiple sources (databases, APIs, flat files).
• Build and optimize batch and real-time data processing workflows.
• Perform complex data transformations and data manipulation.
• Write advanced SQL for efficient data extraction and analysis.
• Work with nested JSON data and large-scale datasets.
• Develop Pandas DataFrames and NumPy arrays for analysis.
• Collaborate with data scientists and analysts to ensure data quality.
• Optimize Spark jobs for performance and scalability.

Skills

PySpark development

Python

NumPy

Pandas

Advanced SQL queries

Distributed data processing

ETL pipelines

Tools

Apache Iceberg

Apache Airflow

AWS

GCP

Job description

Team: Engineering
Experience: 4 - 6 Years
Location: Bangalore
Compensation: As per industry standards

About the Role

We are seeking a skilled PySpark Data Engineer to join our growing data team. In this role, you will design and build scalable data pipelines, optimize distributed data processing, and implement efficient ETL workflows using PySpark and cloud-native technologies.

You will work closely with data scientists, analysts, and engineering teams to ensure reliable data ingestion, transformation, and delivery across sophisticated data platforms and applications.

This is an individual contributor role suited for engineers who are passionate about building high-performance data products and solving complex data engineering challenges.

Key Responsibilities

• Design, develop, and maintain scalable data pipelines using PySpark.
• Fetch and ingest data from multiple data sources such as databases, APIs, and flat files.
• Build and optimize batch and real-time data processing workflows.
• Perform complex data transformations and data manipulation.
• Write advanced SQL queries for efficient data extraction and analysis.
• Work with nested JSON data structures and large-scale datasets.
• Develop Pandas DataFrames and NumPy arrays for data analysis.
• Collaborate with cross-functional teams to ensure high data quality and reliability.
• Optimize Spark jobs for performance and scalability.

Must Have Skills

• Strong hands-on experience in PySpark development.
• Advanced programming skills in Python, including NumPy and Pandas.
• Expertise in writing advanced SQL queries.
• Experience building complex ETL pipelines and data transformation workflows.
• Ability to process and manipulate large datasets and nested JSON structures.
• Strong problem-solving skills with experience in distributed data processing systems.

Good to Have Skills

• Knowledge of Apache Iceberg or other open table formats.
• Experience with Apache Airflow for workflow orchestration.
• Experience with cloud platforms such as AWS or GCP.
• Exposure to product development environments or startup ecosystems.