Data Engineer

Recex Pvt Ltd.; Kolkata District; On-site; INR 1,100,000 - 1,900,000; Full time

Job summary

Recex Pvt Ltd. seeks an experienced Data Engineer to design, build, and maintain scalable data pipelines for batch and real-time processing. The role focuses on ETL/ELT workflows, complex SQL, and data warehousing to empower BI and analytics.

You will ingest data via APIs, streams, and files while ensuring quality and governance across datasets. The ideal candidate has 4–5 years of hands-on experience, strong Python skills, and practical knowledge of ClickHouse, AWS, Kafka, and Airflow, with a

Qualifications

• 4-5 years of hands-on data engineering experience.
• Strong Python proficiency for data workflows.
• Advanced SQL skills for complex queries.
• Experience building ETL/ELT pipelines and data warehousing.

Responsibilities

• Design, develop, and maintain scalable data pipelines for batch and real-time processing.
• Build and optimize ETL/ELT workflows to integrate data from multiple sources.
• Develop and optimize complex SQL queries, database schemas, and data models.
• Work with analytical databases and data warehouses to support BI and reporting.
• Implement data ingestion pipelines using APIs, databases, event streams, and files.
• Ensure data quality, consistency, reliability, and governance.
• Optimize pipeline performance, query execution, storage, and infra costs.
• Collaborate with backend engineers, product teams, and business stakeholders.
• Implement monitoring, logging, error handling, and automated recovery for data pipelines.
• Maintain technical documentation and follow engineering best practices.

Skills

Python

SQL

Data modelling

ETL/ELT pipelines

ClickHouse

AWS

Apache Spark

Kafka

Airflow

Docker

Git & CI/CD

Parquet/lakehouse

Tools

MySQL

PostgreSQL

ClickHouse

Airflow

Job description

Role Overview

We are looking for an experienced Data Engineer with 4-5 years of hands-on experience in building scalable data pipelines, managing large datasets, and designing efficient data architectures.

Key Responsibilities

• Design, develop, and maintain scalable data pipelines for batch and real-time processing.


• Build and optimize ETL/ELT workflows to integrate data from multiple sources.


• Develop and optimize complex SQL queries, database schemas, and data models.


• Work with analytical databases and data warehouses to support business intelligence and reporting.


• Implement data ingestion pipelines using APIs, databases, event streams, and files.


• Ensure data quality, consistency, reliability, and governance.


• Optimize pipeline performance, query execution, storage, and infrastructure costs.


• Collaborate with backend engineers, product teams, and business stakeholders.


• Implement monitoring, logging, error handling, and automated recovery for data pipelines.


• Maintain technical documentation and follow engineering best practices.

Required Skills

• Experience: 4-5 years in data engineering or a similar role.


• Programming: Strong proficiency in Python.


• Databases: Advanced SQL and hands-on experience with MySQL or PostgreSQL.


• Data Processing: Experience building ETL/ELT pipelines and data transformation workflows.


• Analytical Databases: Hands-on experience with ClickHouse.


• Cloud: Practical experience with AWS services such as S3, Glue, Lambda, Athena, or EMR.


• Big Data: Working knowledge of Apache Spark or PySpark.


• Streaming: Understanding of Apache Kafka or other event-streaming technologies.


• Orchestration: Experience with Apache Airflow or similar workflow orchestration tools.


• DevOps: Familiarity with Docker, Git, CI/CD, and monitoring tools.


• Strong understanding of data modelling, partitioning, indexing, and query optimization.

Preferred Qualifications

• Experience handling high-volume transactional and analytical datasets.


• Experience with CDC (Change Data Capture) and incremental data loading.


• Familiarity with lakehouse architecture and data formats such as Parquet.


• Experience building near-real-time analytics and reporting platforms.


• Knowledge of data security, access control, and data governance.

What We Expect

• Strong problem-solving and analytical skills.


• Ability to independently own data pipelines from design to production.


• Focus on reliability, scalability, performance, and maintainability.


• Ability to work effectively with cross-functional engineering teams.


• A proactive approach to identifying and resolving data quality and performance issues.

Success Metrics

• Reliability and freshness of data pipelines.


• Reduction in data processing time and infrastructure costs.


• Accuracy and consistency of analytical datasets.


• Performance and scalability of data infrastructure.


• Timely delivery of business-critical data requirements.