Senior Data Engineer

Algoleap Technologies; Bengaluru; On-site; INR 1,200,000 - 1,800,000; Full time

Job summary

Algoleap Technologies in Bengaluru is seeking an experienced Data Engineer to design and deliver scalable data platforms that power business analytics.

You will build batch and streaming pipelines using Spark, SQL, and AWS Glue, and collaborate with data, analytics, and product teams. This role emphasizes data governance, security, and performance in cloud-native environments.

Qualifications

• Bachelor's or Master's degree in Computer Science, Engineering, IT, or related field.
• 8+ years of experience in Data Engineering or Data Platform development.
• Proven experience building enterprise-scale data platforms and cloud-native data solutions.

Responsibilities

• Design, develop, and maintain scalable data products and engineering solutions.
• Build batch and real-time data pipelines for enterprise platforms.
• Develop and optimize data architectures for scalability, reliability, and performance.
• Support data lakes, data warehouses, lakehouses, and modern analytical ecosystems.
• Collaborate with Data Science, Analytics, Product, and Business teams to gather requirements.
• Provide technical guidance for enterprise data initiatives.

Skills

Data Pipeline Development

ETL/ELT Frameworks

Data Warehousing

Data Lakes & Lakehouses

Data Modeling

Data Architecture

Education

Bachelor's or Master’s degree in Computer Science/Engineering/IT

Job description

Key Responsibilities

Data Engineering & Platform Development

• Design, develop, and maintain scalable data products and engineering solutions.
• Build robust batch and real-time data pipelines for enterprise data platforms.
• Develop and optimize data architectures, ensuring scalability, reliability, and performance.
• Support data lakes, data warehouses, lakehouses, and modern analytical ecosystems.

Data Pipeline & Integration

• Develop and maintain data ingestion frameworks from multiple structured and unstructured data sources.
• Build and manage ETL/ELT pipelines for enterprise analytics and reporting.
• Implement automated data processing workflows and orchestration frameworks.
• Ensure high-quality, secure, and governed data movement across platforms.

Data Modeling & Transformation

• Design and implement scalable data models aligned with business requirements.
• Develop transformation frameworks using SQL, Spark, dbt, AWS Glue, and Airflow.
• Implement data modeling concepts including SCD, schema evolution, and dimensional modeling.
• Optimize data processing performance through tuning and troubleshooting.

Cloud & Data Infrastructure

• Design and operate cloud-based data platforms and infrastructure.
• Support data storage, governance, security, and observability requirements.
• Implement scalable solutions across AWS, Azure, or GCP environments.
• Contribute to platform modernization and cloud transformation initiatives.

Streaming & Advanced Data Processing

• Build and support streaming architectures using Kafka, Flink, or equivalent technologies.
• Develop real-time data processing solutions for operational and analytical use cases.
• Optimize streaming performance, monitoring, and reliability.

DevOps & Automation

• Implement CI/CD pipelines and automated deployment frameworks.
• Follow DevOps best practices for code management, testing, and release management.
• Support infrastructure automation and operational excellence initiatives.

Stakeholder Collaboration

• Partner with Data Science, Analytics, Product, and Business teams to gather requirements.
• Translate business needs into scalable technical solutions.
• Provide technical guidance and support for enterprise data initiatives.

Mandatory Skills

Data Engineering

• Data Pipeline Development
• ETL / ELT Frameworks
• Data Warehousing
• Data Lakes & Lakehouses
• Data Modeling
• Data Architecture

Programming & Query Languages

• Python
• SQL (Expert Level)
• Java / Scala (Preferred)

Big Data & Processing Frameworks

• Apache Spark
• Spark Streaming
• Performance Tuning & Debugging

Data Streaming

• Apache Kafka
• Apache Flink
• Real-Time Data Processing

Cloud Platforms

• AWS (Preferred)
• Azure
• GCP

Data Integration & Orchestration

• AWS Glue
• Airflow
• dbt
• Data Ingestion Frameworks

Data Storage & Formats

• Parquet
• Iceberg
• Data Lake Storage Technologies

DevOps & CI/CD

• Git
• CI/CD Pipelines
• Automated Deployment Frameworks
• DevOps Best Practices

Data Governance & Security

• Data Quality Management
• Data Governance
• Data Privacy & Security
• Metadata Management
• Data Observability

Preferred Skills

• Modern Data Platform Architecture
• Lakehouse Implementations
• Cloud-Native Data Engineering
• Enterprise Analytics Platforms
• Advanced Performance Optimization
• Data Product Development
• Infrastructure as Code (IaC)

Education & Experience

• Bachelors or Master’s degree in Computer Science, Engineering, Information Technology, or a related field.
• 8+ years of experience in Data Engineering, Data Platform Development, or related areas.
• Proven experience building enterprise-scale data platforms and cloud-native data solutions.