Data Engineer

Nineleaps; Bengaluru; Presencial; INR 1.500.000 - 2.500.000; Jornada completa

Descripción de la vacante

Nineleaps in Bengaluru is seeking a Data Engineer with 3-4 years of experience to design, build, and optimize data pipelines using Python, PySpark, and SQL. You will work with large datasets on Hadoop/Spark ecosystems and cloud platforms.

You will collaborate with data analysts and scientists, ensure data quality, and tune performance of ETL/ELT jobs. Strong communication and independence are valued.

Formación

• 3-4 years of relevant experience in data engineering.
• Strong problem-solving and analytical skills.
• Ability to work independently on data engineering tasks.
• Good communication and collaboration skills.

Responsabilidades

• Design, develop, and maintain scalable data pipelines for processing structured and unstructured data.
• Develop data transformation and processing workflows using Python, PySpark, and SQL.
• Work with large datasets using Apache Spark and related big data technologies.
• Build and maintain ETL/ELT pipelines to extract, transform, and load data from multiple sources.
• Work with data storage and processing technologies such as Hive, Hadoop, and cloud-based data platforms.
• Optimize SQL queries and data processing jobs for improved performance.
• Perform data validation, troubleshoot pipeline failures, and ensure data quality.
• Collaborate with data analysts, data scientists, and other engineering teams to support business requirements.
• Monitor scheduled jobs and ensure the reliability and efficiency of data pipelines.

Conocimientos

Python

SQL

PySpark

ETL/ELT

Big Data

Data Processing

Airflow

Cloud Platforms

Git

Herramientas

Hive

Hadoop

Spark SQL

Airflow

Azure Databricks

Azure Data Factory

Descripción del empleo

We are looking for a skilled Data Engineer with 3-4 years of experience in developing, maintaining, and optimizing data pipelines and data processing solutions. The ideal candidate should have strong hands‑on experience in Python, SQL, PySpark, and big data technologies, along with exposure to cloud‑based data platforms.

Responsibilities:

• Design, develop, and maintain scalable data pipelines for processing structured and unstructured data.
• Develop data transformation and processing workflows using Python, PySpark, and SQL.
• Work with large datasets using Apache Spark and related big data technologies.
• Build and maintain ETL/ELT pipelines to extract, transform, and load data from multiple sources.
• Work with data storage and processing technologies such as Hive, Hadoop, and cloud‑based data platforms.
• Optimize SQL queries and data processing jobs for improved performance.
• Perform data validation, troubleshoot pipeline failures, and ensure data quality.
• Collaborate with data analysts, data scientists, and other engineering teams to support business requirements.
• Monitor scheduled jobs and ensure the reliability and efficiency of data pipelines.

Requirements:

• 3-4 years of relevant experience in data engineering.
• Strong problem‑solving and analytical skills.
• Ability to work independently on data engineering tasks.
• Good communication and collaboration skills.

Required Skills:

• Python: Strong hands‑on experience in Python programming and data processing.
• SQL: Good knowledge of complex SQL queries, joins, aggregations, and query optimization.
• PySpark / Apache Spark: Experience in distributed data processing and large‑scale data transformations.
• ETL/ELT: Hands‑on experience building and maintaining data pipelines.
• Big Data Technologies: Exposure to Hive, Hadoop, Spark SQL, or similar technologies.
• Data Processing: Understanding of data warehousing concepts, data modeling, and file formats such as Parquet and JSON.
• Workflow Orchestration: Exposure to Apache Airflow or similar scheduling tools.
• Cloud Platforms: Exposure to Azure, AWS, or GCP is preferred.
• Version Control: Working knowledge of Git.

Good to Have:

• Experience with Azure Databricks, Azure Data Factory, or similar cloud data engineering services.
• Familiarity with data lake and data warehouse architectures.
• Experience with performance tuning and optimizing Spark jobs.
• Understanding of data quality checks and pipeline monitoring.