Data Engineer

Recex; Kolkata District; On-site; INR 1,200,000 - 2,000,000; Full time

Job summary

Recex is seeking an experienced Data Engineer to design and maintain scalable data pipelines for batch and real-time processing, integrating data from multiple sources and ensuring quality and governance. You will work with analytics databases and data warehouses to support BI and reporting, using Python, SQL, Spark, Kafka, and Airflow in an AWS environment.

This role requires 4–5 years in data engineering, strong SQL, and hands-on with ClickHouse, MySQL/PostgreSQL, and lakehouse concepts.

Qualifications

• 4–5 years of data engineering or equivalent experience.
• Proficient in Python.
• Advanced SQL skills; MySQL or PostgreSQL experience.
• ETL/ELT pipeline development and data transformation.
• Experience with ClickHouse.
• Hands-on with AWS (S3, Glue, Lambda, Athena, EMR).
• Familiar with Apache Spark or PySpark.
• Knowledge of Apache Kafka or other streaming tech.
• Experience with Airflow or similar workflow orchestration.
• Familiar with Docker, Git, CI/CD, and monitoring tools.
• Strong data modelling, partitioning, indexing, and query optimization.

Responsibilities

• Design, develop, and maintain scalable data pipelines for batch and real-time processing.
• Build and optimize ETL/ELT workflows.
• Develop SQL queries, schemas, and data models.
• Work with analytical databases and data warehouses to support BI and reporting.
• Implement data ingestion pipelines using APIs, databases, event streams, and files.
• Ensure data quality, consistency, reliability, and governance.
• Optimize pipeline performance, query execution, storage, and infrastructure costs.
• Collaborate with backend engineers, product teams, and business stakeholders.
• Implement monitoring, logging, error handling, and automated recovery for data pipelines.
• Maintain technical documentation and follow engineering best practices.

Skills

Python

ETL/ELT pipelines

SQL

Data modeling

Data governance

Tools

ClickHouse

Apache Spark

PySpark

Apache Airflow

Apache Kafka

Docker

Git

CI/CD

AWS S3

AWS Glue

AWS Lambda

AWS Athena

AWS EMR

Job description

Role Overview

We are looking for an experienced Data Engineer with 45 years of hands-on experience in building scalable data pipelines, managing large datasets, and designing efficient data architectures.Key Responsibilities

• Design, develop, and maintain scalable data pipelines for batch and real-time processing.
• Build and optimize ETL/ELT workflows to integrate data from multiple sources.
• Develop and optimize complex SQL queries, database schemas, and data models.
• Work with analytical databases and data warehouses to support business intelligence and reporting.
• Implement data ingestion pipelines using APIs, databases, event streams, and files.
• Ensure data quality, consistency, reliability, and governance.
• Optimize pipeline performance, query execution, storage, and infrastructure costs.
• Collaborate with backend engineers, product teams, and business stakeholders.
• Implement monitoring, logging, error handling, and automated recovery for data pipelines.
• Maintain technical documentation and follow engineering best practices.

Required Skills

• Experience: 4–5 years in data engineering or a similar role.
• Programming: Strong proficiency in Python.
• Databases: Advanced SQL and hands-on experience with MySQL or PostgreSQL.
• Data Processing: Experience building ETL/ELT pipelines and data transformation workflows.
• Analytical Databases: Hands-on experience with ClickHouse.
• Cloud: Practical experience with AWS services such as S3, Glue, Lambda, Athena, or EMR.
• Big Data: Working knowledge of Apache Spark or PySpark.
• Streaming: Understanding of Apache Kafka or other event-streaming technologies.
• Orchestration: Experience with Apache Airflow or similar workflow orchestration tools.
• DevOps: Familiarity with Docker, Git, CI/CD, and monitoring tools.
• Strong understanding of data modelling, partitioning, indexing, and query optimization.

Preferred Qualifications

• Experience handling high-volume transactional and analytical datasets.
• Experience with CDC (Change Data Capture) and incremental data loading.
• Familiarity with lakehouse architecture and data formats such as Parquet.
• Experience building near-real-time analytics and reporting platforms.
• Knowledge of data security, access control, and data governance.

What We Expect

• Strong problem-solving and analytical skills.
• Ability to independently own data pipelines from design to production.
• Focus on reliability, scalability, performance, and maintainability.
• Ability to work effectively with cross-functional engineering teams.
• A proactive approach to identifying and resolving data quality and performance issues.

Success Metrics

• Reliability and freshness of data pipelines.
• Reduction in data processing time and infrastructure costs.
• Accuracy and consistency of analytical datasets.
• Performance and scalability of data infrastructure.
• Timely delivery of business-critical data requirements.