Data Engineer

Indus Net Technologies Limited; Kolkata District; On-site; INR 850,000 - 1,300,000; Full time

Job summary

Indus Net Technologies Limited is seeking a Data Engineer with 2–3 years of experience to design and develop scalable data pipelines and data lakehouse solutions in a cloud environment.

You will collaborate with Data Architects, BI Developers, Data Scientists and PMs to translate business requirements into production-ready data solutions, spanning Bronze to Gold data layers and data quality checks.

Qualifications

• Bachelor's degree in Computer Science, Information Technology, Engineering, Data Engineering or a related technical discipline.
• 2–3 years of professional experience in Data Engineering.
• Strong hands-on experience with SQL.
• Good programming experience in Python.
• Experience developing and supporting ETL/ELT pipelines.
• Understanding of Data Lake, Lakehouse or modern data platform architectures.
• Experience working with structured and semi-structured data.
• Good understanding of data transformation, cleansing, validation and data quality concepts.
• Experience working with relational databases.
• Understanding of dimensional modelling and analytical data structures.
• Good understanding of data engineering best practices including logging, error handling, testing and monitoring.
• Experience working in an Agile development environment.
• Ability to understand business requirements and translate them into technical data solutions.

Responsibilities

• Design, develop and maintain scalable ETL/ELT data pipelines.
• Build data ingestion and transformation pipelines across databases, APIs, enterprise applications and file-based sources.
• Develop data processing solutions using SQL, Python and PySpark/Spark.
• Work with cloud data platforms, Data Lakes and Lakehouse architectures.
• Develop and maintain Bronze, Silver and Gold/curated data layers.
• Implement data transformations, cleansing, validation and data quality checks.
• Work with Databricks, Microsoft Fabric and/or Azure data services.
• Develop efficient and reusable data engineering frameworks and components.
• Optimize SQL queries, Spark jobs and data pipelines for performance and cost.
• Implement appropriate error handling, logging, monitoring and pipeline recovery mechanisms.
• Support data modelling for analytical and reporting use cases.
• Collaborate with BI teams to provide trusted and analytics-ready datasets.
• Support Data Scientists and AI teams with appropriately prepared datasets and feature-ready data.
• Follow engineering practices around Git, version control, CI/CD, testing and deployment.
• Contribute to technical documentation, data lineage and operational runbooks.
• Troubleshoot production data pipeline and data quality issues.
• Participate in code reviews and follow development standards.
• Work in an Agile project environment and collaborate with distributed teams and client stakeholders.

Skills

SQL

Python

PySpark/Spark

Azure

Databricks

REST APIs

CI/CD

Power BI

Education

Bachelor's degree in CS/IT/Engineering

Tools

Azure Data Factory

Microsoft Fabric

Delta Lake

Git

Job description

About the Role

We are looking for a Data Engineer with 2–3 years of experience to join our Data & Analytics team and contribute to the development of modern, cloud-based data platforms and analytics solutions.

The role involves designing and developing scalable data pipelines, data lake/lakehouse solutions, data transformations and curated data layers that support Business Intelligence, Advanced Analytics, AI/ML and Decision Intelligence use cases.

The engineer will work closely with Data Architects, Business Analysts, BI Developers, Data Scientists and Project Managers to convert business requirements into reliable and production-ready data solutions.

Key Responsibilities

Design, develop and maintain scalable ETL/ELT data pipelines.

Build data ingestion and transformation pipelines across databases, APIs, enterprise applications and file-based sources.

Develop data processing solutions using SQL, Python and PySpark/Spark.

Work with cloud data platforms, Data Lakes and Lakehouse architectures.

Develop and maintain Bronze, Silver and Gold/curated data layers.

Implement data transformations, cleansing, validation, reconciliation and data quality checks.

Work with Databricks, Microsoft Fabric and/or Azure data services.

Develop efficient and reusable data engineering frameworks and components.

Optimize SQL queries, Spark jobs and data pipelines for performance and cost.

Implement appropriate error handling, logging, monitoring and pipeline recovery mechanisms.

Support data modelling for analytical and reporting use cases.

Collaborate with BI teams to provide trusted and analytics-ready datasets.

Support Data Scientists and AI teams with appropriately prepared datasets and feature-ready data.

Follow engineering practices around Git, version control, CI/CD, testing and deployment.

Contribute to technical documentation, data lineage and operational runbooks.

Troubleshoot production data pipeline and data quality issues.

Participate in code reviews and follow development standards.

Work in an Agile project environment and collaborate with distributed teams and client stakeholders.

Typical Technology Environment

The team works across modern cloud and data technologies including:

Microsoft Fabric

Databricks

SQL

Python

Data Lakes / Lakehouses

Azure Data Factory / Fabric Data Factory

Power BI

Relational and NoSQL databases

APIs and enterprise data sources

The exact technology stack may vary by project.

Requirements

Requirements

Required

Bachelor's degree in Computer Science, Information Technology, Engineering, Data Engineering or a related technical discipline.

2–3 years of professional experience in Data Engineering.

Strong hands-on experience with SQL.

Good programming experience in Python.

Experience developing and supporting ETL/ELT pipelines.

Understanding of Data Lake, Lakehouse or modern data platform architectures.

Experience working with structured and semi-structured data.

Good understanding of data transformation, cleansing, validation and data quality concepts.

Experience working with relational databases.

Understanding of dimensional modelling and analytical data structures.

Good understanding of data engineering best practices including logging, error handling, testing and monitoring.

Experience working in an Agile development environment.

Ability to understand business requirements and translate them into technical data solutions.

Preferred

Hands-on experience with Microsoft Azure.

Experience with Databricks.

Experience with Apache Spark / PySpark.

Experience with Microsoft Fabric.

Experience with Azure Data Factory or Fabric Data Factory.

Experience with Delta Lake / Delta Tables.

Exposure to Apache Iceberg or other open table formats.

Experience with Git and CI/CD.

Experience with Power BI or other BI platforms.

Experience integrating data from enterprise applications such as SAP, Salesforce, ERP, CRM, LIMS or similar systems.

Exposure to REST APIs and API-based data ingestion.

Exposure to streaming/event-driven data pipelines.

Understanding of data governance, security, access control and PII handling.

Exposure to AI/ML data pipelines or analytics platforms.

Education

Bachelor's degree in Computer Science, Information Technology, Engineering or a related technical field.

Experience

2–3 years of relevant professional Data Engineering experience.

Candidates with strong hands-on project experience in SQL, Python, Spark and cloud data engineering may also be considered where their experience is slightly outside the stated range.