Data Engineer-Apache Iceberg

Pi Square Technologies; Pune District, Chennai District, Bengaluru; Hybrid; INR 2,500,000 - 3,800,000; Full time

Job summary

Pi Square Technologies is seeking an experienced Technology Lead to architect and deliver data engineering solutions using Databricks, Apache Iceberg, AWS, and PySpark for Banking/Financial Services clients. You will lead design and development of scalable data platforms in an Agile environment.

The ideal candidate has 7+ years of hands-on Big Data and cloud experience, strong coding skills in Python and SQL, and a proven track record of delivering secure, governance-compliant data lakes.

Qualifications

• 7+ years of experience in technology leadership for Big Data and cloud engineering.
• Strong expertise in Databricks, AWS, Big Data, and PySpark.
• Hands-on experience with Databricks, Iceberg, Spark, PySpark, and SQL.

Responsibilities

• Design, develop, and maintain scalable data engineering solutions using Databricks, Iceberg, AWS, PySpark, and Spark.
• Build and optimize batch and real-time data pipelines to support analytics.
• Develop ETL/ELT using Python and SQL.
• Design and implement cloud-native data architectures using AWS services.
• Translate requirements into robust technical solutions with best practices.
• Collaborate with stakeholders and cross-functional teams to resolve data challenges.
• Perform code reviews, troubleshooting, and optimization of pipelines and workloads.
• Ensure data security, governance, and access management.

Skills

Databricks & Big Data

Databricks

Apache Iceberg

Apache Spark

PySpark

Spark SQL

Data Lake / Lakehouse Architecture

Tools

AWS Glue

AWS CloudFormation

AWS Lake Formation

AWS IAM

AWS Lambda

Amazon Redshift

Job description

Job Location- Pune, Chennai, Bangalore

Must have skills which are non-negotiable

1. Databricks,
2. Apache iceberg
3. Pyspark
4. AWS Core Data Services

Role: Technology Lead Databricks, AWS & Big Data

Experience: 7+ Years

Domain: Banking / Financial Services

Location: Offshore(Pune, Chennai and Bangalore) / Hybrid

Job Summary

We are seeking an experienced Technology Lead with strong expertise in Databricks, AWS, Big Data, and PySpark development to design, develop, and deliver scalable data engineering solutions for

Banking and Financial Services clients.

The ideal candidate will have hands-on experience building

modern data platforms using Databricks, Apache Iceberg, AWS cloud services, and Spark-based technologies while leading development activities in an Agile environment.

Key Responsibilities

• Design, develop, and maintain scalable data engineering solutions using Databricks, Apache Iceberg, AWS, PySpark, and Spark.
• Build and optimize batch and real-time data processing pipelines to support business analytics and reporting requirements.
• Develop highly efficient ETL/ELT solutions using Python, SQL, and PySpark.
• Design and implement cloud-native data architectures leveraging AWS services such as Glue, Lake Formation, CloudFormation, IAM, Lambda, and Redshift.
• Translate business and functional requirements into robust technical solutions while ensuring adherence to quality standards and best practices.
• Collaborate with business stakeholders, architects, data analysts, and cross-functional teams to identify and resolve data-related challenges.
• Perform code reviews, troubleshooting, performance tuning, and optimization of data pipelines and Spark workloads.
• Ensure data security, governance, and access management using AWS and Databricks best practices.
• Support production deployments, incident resolution, root cause analysis, and continuous process improvement initiatives.
• Stay current with emerging technologies, industry trends, and best practices in Big Data and Cloud Engineering.

Required Technical Skills

• Databricks & Big Data
• Databricks
• Apache Iceberg
• Apache Spark
• PySpark
• Spark SQL
• Data Lake / Lakehouse Architecture

AWS Cloud Services

• AWS Glue
• AWS CloudFormation
• AWS Lake Formation

AWS IAM

AWS Lambda

Amazon Redshift

Programming Languages

• Python
• SQL Data Engineering
• ETL / ELT Development
• Data Warehousing

• Data Integration
• Data Modeling

Performance Optimization --- Agile & Collaboration Skills

Experience working in Agile/Scrum environments.

Active participation in Scrum ceremonies.

Important :

Notice period : Immediate to 15-30 days ONLY.

Candidates serving Notice period will be preferred.

Please note - There should not be Career Gap more than 6 months(No Career Break)

Graduation score should be 60% & above

Pre Screening BGV (Background Verification)is mandatory [UAN/PF without any overlap issue is required.]