Data Engineer

InSync Analytics Corp.; Mumbai; Hybrid; INR 2,500,000 - 4,500,000; Full time

Job summary

InSync Analytics Corp. in Mumbai is seeking a Senior Data Engineer to design and operate robust data pipelines using our modern data stack.

You will build real-time streaming pipelines with Kafka and CDC, develop batch workflows, and implement data quality monitoring. You will architect and maintain data lake infrastructure with Apache Iceberg, optimize storage across PostgreSQL, Snowflake, and ClickHouse, and collaborate with cross-functional teams while mentoring junior engineers.

Qualifications

• Hands-on experience with Iceberg in production environments.
• Experience building real-time pipelines with Kafka and CDC.
• Familiarity with data mesh or federated architectures.
• Knowledge of data quality tooling (Great Expectations, Deequ).

Responsibilities

• Design and implement robust, scalable data pipelines using our modern data stack.
• Build real-time streaming pipelines with Kafka and Change Data Capture (CDC).
• Develop batch processing workflows for data transformation and analytics.
• Implement data quality frameworks and monitoring systems.
• Architect and maintain our data lake infrastructure using Apache Iceberg.
• Optimize data storage and retrieval across multiple data stores (PostgreSQL, Snowflake, ClickHouse).
• Design partitioning and sharding strategies for high-volume datasets.
• Collaborate with data scientists, analysts, and product teams; mentor junior engineers; participate in code reviews; maintain high engineering standards.
• Document architectures, processes, and best practices.

Skills

Cloud platforms (AWS/GCP/Azure)

Data engineering practices

Mentoring engineers

Agile development

Tools

Apache Iceberg

Kafka

Debezium

PostgreSQL

Snowflake

ClickHouse

Great Expectations/Deequ

Data mesh architectures

Job description

Role & responsibilities

Data Pipeline Development

• Design and implement robust, scalable data pipelines using our modern data stack
• Build real-time streaming pipelines with Kafka and Change Data Capture (CDC)
• Develop batch processing workflows for data transformation and analytics
• Implement data quality frameworks and monitoring systems

Infrastructure & Architecture

• Architect and maintain our data lake infrastructure using Apache Iceberg
• Optimize data storage and retrieval across multiple data stores (PostgreSQL, Snowflake, ClickHouse)
• Design partitioning and sharding strategies for high-volume datasets.

Data Integration & CDC

• Set up and maintain Change Data Capture (CDC) pipelines using Debezium
• Build real-time data synchronization between operational and analytical systems
• Handle schema evolution and ensure backward compatibility
• Implement idempotent data processing patterns

Performance Optimization

• Monitor and troubleshoot performance bottlenecks
• Optimize query performance across different data engines
• Implement efficient data compression and storage strategies
• Design and maintain data indexing strategies

Collaboration & Leadership

• Collaborate with data scientists, analysts, and product teams
• Mentor junior engineers and contribute to technical decision-making
• Participate in code reviews and maintain high engineering standards
• Document architectures, processes, and best practices

Preferred candidate profile

Advanced Technical Experience

Experience with multiple cloud providers (AWS, GCP, Azure)

Hands-on experience with Apache Iceberg in production environments

Experience with data mesh or federated data architectures

Knowledge of machine learning pipelines and feature stores

Experience with data quality tools (Great Expectations, Deequ, etc.)

Leadership & Collaboration

Experience mentoring junior engineers or leading technical initiatives

Strong communication skills with both technical and non-technical stakeholders

Experience working in agile development environments

Track record of improving data engineering practices and standards

Industry Experience

Experience in relevant industry: fintech, e-commerce, SaaS, etc.

Understanding of regulatory requirements (GDPR, SOX, PCI, etc.)

Experience with A/B testing and experimentation platforms