Data Engineer

JobItUs; Kolkata District; Sur place; INR 1 200 000 - 2 400 000; Plein temps

Résumé du poste

JobItUs in India is seeking an experienced Data Engineer with 4–5 years of hands-on experience to design scalable data pipelines and efficient data architectures.

You will build and optimize ETL/ELT workflows, work with batch and real-time data, develop complex SQL, and partner with product and backend teams to enable robust analytics.

The role also focuses on data quality, governance, monitoring, and cost optimization across cloud environments.

Qualifications

• Able to design scalable data pipelines and architectures.
• Strong SQL skills and data modelling knowledge.
• Experience with batch and real-time data processing.

Responsabilités

• Design, develop, and maintain scalable data pipelines for batch and real-time processing.
• Build and optimize ETL/ELT workflows to integrate data from multiple sources.
• Develop and optimize complex SQL queries, database schemas, and data models.
• Work with analytical databases and data warehouses to support BI and reporting.
• Implement data ingestion pipelines using APIs, databases, event streams, and files.
• Ensure data quality, consistency, reliability, and governance.
• Optimize pipeline performance, query execution, storage, and infrastructure costs.
• Collaborate with backend engineers, product teams, and business stakeholders.
• Implement monitoring, logging, error handling, and automated recovery for data pipelines.
• Maintain technical documentation and follow engineering best practices.

Connaissances

Python

SQL

ETL/ELT

Airflow

Spark

Kafka

PostgreSQL

MySQL

ClickHouse

AWS

Parquet

Formation

Bachelor's degree in Computer Science

Outils

Docker

Git

CI/CD

Description du poste

Role Overview

We are looking for an experienced Data Engineer with 4-5 years of hands-on experience in building scalable data pipelines, managing large datasets, and designing efficient data architectures.

Key Responsibilities

• Design, develop, and maintain scalable data pipelines for batch and real-time processing.


• Build and optimize ETL/ELT workflows to integrate data from multiple sources.


• Develop and optimize complex SQL queries, database schemas, and data models.


• Work with analytical databases and data warehouses to support business intelligence and reporting.


• Implement data ingestion pipelines using APIs, databases, event streams, and files.


• Ensure data quality, consistency, reliability, and governance.


• Optimize pipeline performance, query execution, storage, and infrastructure costs.


• Collaborate with backend engineers, product teams, and business stakeholders.


• Implement monitoring, logging, error handling, and automated recovery for data pipelines.


• Maintain technical documentation and follow engineering best practices.

Required Skills

• Experience: 4-5 years in data engineering or a similar role.


• Programming: Strong proficiency in Python.


• Databases: Advanced SQL and hands-on experience with MySQL or PostgreSQL.


• Data Processing: Experience building ETL/ELT pipelines and data transformation workflows.


• Analytical Databases: Hands-on experience with ClickHouse.


• Cloud: Practical experience with AWS services such as S3, Glue, Lambda, Athena, or EMR.


• Big Data: Working knowledge of Apache Spark or PySpark.


• Streaming: Understanding of Apache Kafka or other event-streaming technologies.


• Orchestration: Experience with Apache Airflow or similar workflow orchestration tools.


• DevOps: Familiarity with Docker, Git, CI/CD, and monitoring tools.


• Strong understanding of data modelling, partitioning, indexing, and query optimization.

Preferred Qualifications

• Experience handling high-volume transactional and analytical datasets.


• Experience with CDC (Change Data Capture) and incremental data loading.


• Familiarity with lakehouse architecture and data formats such as Parquet.


• Experience building near-real-time analytics and reporting platforms.


• Knowledge of data security, access control, and data governance.

What We Expect

• Strong problem-solving and analytical skills.


• Ability to independently own data pipelines from design to production.


• Focus on reliability, scalability, performance, and maintainability.


• Ability to work effectively with cross-functional engineering teams.


• A proactive approach to identifying and resolving data quality and performance issues.

Success Metrics

• Reliability and freshness of data pipelines.


• Reduction in data processing time and infrastructure costs.


• Accuracy and consistency of analytical datasets.


• Performance and scalability of data infrastructure.


• Timely delivery of business-critical data requirements.