Data Engineer

Calsoft; Pune District; On-site; INR 2,800,000 - 4,500,000; Full time

Job summary

Calsoft in Pune is seeking a Senior Data Engineer to design, develop, and optimize scalable cloud-based data platforms that power analytics, operational intelligence, and enterprise applications.

The ideal candidate will build reliable data pipelines, design modern data architectures, and enable data-driven decision making through high-quality, secure, and scalable solutions. Strong Python and AWS expertise are essential.

Qualifications

• Strong Python programming skills and production-grade data engineering experience.
• Experience building scalable AWS-based data platforms for analytics.
• Solid SQL knowledge and data modeling with batch/streaming processing.

Responsibilities

• Design, develop, and maintain scalable data pipelines on AWS.
• Build ETL/ELT frameworks to ingest data from multiple sources.
• Develop reusable data services for analytics, reporting, and ML workloads.
• Optimize data models, data quality, lineage, and governance.
• Automate deployment and monitoring using IaC and CI/CD practices.
• Collaborate with data scientists, software engineers, and product owners.

Skills

Python

AWS

Data pipelines

SQL

ETL/ELT

Spark

Tools

Apache Spark

Airflow

Terraform

Job description

Senior Data Engineer (AWS | Python | Data Platform)

Location: Pune (5 Days / Week)

Role Summary:

We are looking for a Senior Data Engineer to design, develop, and optimize scalable cloud-based data platforms that power analytics, operational intelligence, and enterprise applications. The ideal candidate is passionate about building reliable data pipelines, designing modern data architectures, and enabling data-driven decision making through high-quality, secure, and scalable solutions.

Key Responsibilities:

• Data Engineering & Platform Development Design, develop, and maintain scalable batch and streaming data pipelines on AWS.
• Build robust ETL/ELT frameworks for ingesting data from multiple internal and external sources.
• Develop reusable data services that support analytics, reporting, machine learning, and operational workloads.
• Design efficient data models optimized for both analytical and operational use cases. Ensure data quality, consistency, lineage, and governance across the data platform.
• Cloud & Infrastructure Build and optimize cloud-native data infrastructure using AWS services.
• Design highly available, fault-tolerant, and scalable data processing architectures.
• Automate deployment, monitoring, and operational workflows using Infrastructure as Code and CI/CD practices.
• Optimize storage, compute utilization, and overall cloud costs.
• Performance & Reliability Improve pipeline performance, scalability, and reliability.
• Monitor production workloads and proactively resolve bottlenecks.
• Optimize SQL queries, data partitioning, indexing strategies, and processing performance.
• Troubleshoot production issues and implement preventive improvements.
• Collaboration Work closely with Data Scientists, Software Engineers, Product Owners, and Architects to understand business requirements.
• Support downstream analytics, dashboards, and reporting solutions.
• Participate in architecture discussions, code reviews, and technical design sessions. Mentor junior engineers and promote engineering best practices.

Required Technical Skills:

• Programming Strong Python programming skills.
• Experience building production-grade data engineering solutions.
• Knowledge of object-oriented design, testing, and clean coding practices.
• AWS Hands-on experience with several of the following: S3 Glue Lambda EMR Athena Redshift RDS Step Functions EventBridge CloudWatch IAM ECS/EKS (preferred) Data Engineering ETL/ELT pipeline development Data warehousing concepts Batch and streaming data processing Data modeling Data validation and quality frameworks Metadata and lineage concepts Databases Advanced SQL PostgreSQL MySQL Redshift Performance tuning and query optimization Development Practices Git CI/CD pipelines Unit testing Docker Agile/Scrum development

Preferred Skills:

• Apache Spark or PySpark Kafka or Kinesis Airflow or AWS Managed Workflows Infrastructure as Code (Terraform or CloudFormation)
• Data Lake architecture Lakehouse concepts
• Experience supporting Machine Learning data pipelines
• Experience with observability and monitoring tools
• Desired Experience 610 years of experience in Data Engineering.
• Experience building cloud-native data platforms on AWS.
• Experience handling large-scale structured and semi-structured datasets.
• Strong understanding of distributed data processing and performance optimization.
• Experience working in enterprise production environments with high availability requirements.