Data Engineer

Allstate Solutions (ASPL); Pune District, Bengaluru; Hybrid; INR 1,200,000 - 1,800,000; Full time

Job summary

Allstate Solutions (ASPL) seeks an experienced Data Engineer to design, build, and operate scalable data pipelines using Apache Spark and cloud-native data technologies. You will ingest, transform, and curate data from diverse sources into analytics-ready datasets and support reporting, dashboards, and ML workloads.

The role emphasizes data modeling, schema management, and delivering robust, production-ready pipelines with CI/CD.

Qualifications

• Strong experience as a Data Engineer building and operating production data pipelines.
• Hands-on experience with Apache Spark for largescale data processing.
• Proficiency in Python, SQL, and data transformation best practices.
• Experience with cloudbased data platforms and storage.
• Familiarity with Microsoft Fabric, One Lake, or similar analytics platforms (strong plus).
• Experience designing and optimizing data models for analytical workloads.
• Understanding of distributed data processing concepts, performance tuning, and fault tolerance.
• Experience with CI/CD, version control, and infrastructureascode concepts.
• Strong problemsolving skills and ability to troubleshoot complex data issues.
• 4+ years of experience in data engineering or equivalent role.

Responsibilities

• Design, build, and maintain scalable data pipelines using Apache Spark and cloud-native data tech.
• Develop and optimize ETL/ELT workflows to ingest and curate data into analytics-ready datasets.
• Implement data modeling and transformation to support reporting and analytics.
• Build data processing workloads within Lakehouse platforms like Microsoft Fabric / One Lake.
• Ensure data quality, reliability, and consistency with validation and monitoring.
• Optimize Spark jobs for performance, cost, and scalability.
• Manage and evolve data schemas amid drift and upstream changes.
• Develop reusable frameworks and patterns to boost productivity.
• Implement CI/CD pipelines for data workloads with automated testing and rollout.
• Monitor pipelines, troubleshoot failures, resolve data issues.
• Collaborate with analytics engineers, BI developers, and data scientists.
• Partner with security and governance teams for data access controls.
• Contribute to Agile processes like sprint planning and reviews.

Skills

Apache Spark

Python

SQL

Data Pipelines

Cloud Platforms

CI/CD

Data Modeling

Lakehouse

Microsoft Fabric

Tools

Microsoft Fabric

One Lake

Job description

Key Responsibilities

• Design, build, and maintain scalable batch and streaming data pipelines using Apache Spark and cloudnative data technologies.


• Develop and optimize ETL/ELT workflows to ingest, transform, and curate data from diverse source systems into analyticsready datasets.


• Implement data modeling and transformation logic to support reporting, dashboards, and downstream analytical and machine learning workloads.


• Build and manage data processing workloads within modern Lakehouse platforms, including Microsoft Fabric / One Lake (preferred).


• Ensure data quality, reliability, and consistency by implementing validation checks, monitoring, and reconciliation processes.


• Optimize Spark jobs for performance, cost efficiency, and scalability across large and complex datasets.


• Manage and evolve data schemas while handling schema drift and upstream source changes.


• Develop reusable frameworks, libraries, and standardized patterns to improve data engineering productivity and consistency.


• Implement CI/CD pipelines for data workloads to enable automated testing, deployment, and rollback.


• Monitor data pipelines and jobs, troubleshoot failures, and resolve performance or data quality issues.


• Partner with analytics engineers, BI developers, and data scientists to understand data requirements and deliver curated datasets.


• Collaborate with platform, security, and governance teams to ensure data security, compliance, and proper access controls.


• Contribute to Agile delivery processes, including sprint planning, design reviews, and continuous improvement initiatives.

Required Qualifications

• Strong experience as a Data Engineer building and operating production data pipelines.


• Handson experience with Apache Spark for largescale data processing.


• Proficiency in Python, SQL, and data transformation best practices.


• Experience with cloudbased data platforms and storage (e.g., Data Lakes, Lakehouse architectures).


• Familiarity with Microsoft Fabric, One Lake, or similar analytics platforms (strong plus).


• Experience designing and optimizing data models for analytical workloads.


• Understanding of distributed data processing concepts, performance tuning, and fault tolerance.


• Experience with CI/CD, version control, and infrastructureascode concepts.


• Strong problemsolving skills and ability to troubleshoot complex data issues.


• Excellent communication skills and ability to collaborate across technical and nontechnical teams.


• 4+ years of experience in data engineering or equivalent role (preferred).

Preferred / NicetoHave Skills

• Experience with realtime or eventdriven data processing.


• Familiarity with data governance, metadata management, and data quality frameworks.


• Exposure to orchestration tools and workflow management systems.


• Experience supporting analytical, reporting, or machine learning use cases.Role & responsibilities