Data Engineer

ValueLabs; Bengaluru; On-site; INR 1,200,000 - 2,400,000; Full time

Job summary

ValueLabs in Bengaluru, Karnataka, seeks an experienced data engineering professional proficient in PySpark, Python, Hadoop, SQL and Airflow to join our data analytics team.

You will design and implement end-to-end ETL pipelines, build data marts, and optimize data workflows across production environments, collaborating with cross-functional teams and delivering robust, maintainable code.

Qualifications

• 5+ years of commercial experience in a data-driven role.
• Hands-on experience building data Marts/ETL pipelines.
• Expert in ETL scripting with Python and PySpark.

Responsibilities

• Develop and maintain end-to-end data pipelines.
• Build and optimize data marts and ETL processes.
• Collaborate with cross-functional teams to deliver robust data solutions.

Skills

PySpark

Python

Hadoop

SQL

Airflow

Tools

Jupyter

Git

Job description

Primary Skill: PySpark, Python, Hadoop, SQL, Airflow

Key responsibilities include:

5+ years of previous commercial experience in a data-driven role, hands-on experience building data Marts/ ETL Pipelines Pyspark Python expert in ETL scripting Ability to write clean, maintainable, and robust code in Python End to END SDLC, build UAT, UAT bug fix, Production deployment activities and postproduction Support Analytical expert in Oracle query and data analysis, also in pyspark code debugging Understanding and expertise of software engineering concepts and best practices pipelines in production and ability to work across, structured, semi-structured and unstructured data Banking Clients/ Banking Domain Knowledge Data Warehousing strong knowledge base.

• Hands-on experience of technologies like Python, Spark/Pyspark, Hadoop/MapReduce/HIVE, Pandas etc.
• Familiarity with query languages and database technologies, CI/CD, testing and validation of data
• Tech stack and activities that you would use and perform on a daily basis:
  • Python
  • Spark (PySpark)
  • Jupyter
  • SQL and No-SQL DBMS
  • Git (as source code versioning and CI/CD)
  • Data Analysis Expert
  • Imputation Techniques
  • Data Linking / Cleansing
  • Feature Engineering
  • Apache Airflow/ Oozie
  • Jenkins Pipeline knowledge
• Collaborative - able to coordinate with multiple teams and resolve the dependencies and deliver the end goal.
• Excellent interpersonal, and communication skills
• Ability to communicate effectively with technical and non-technical audience
• Ability to work under pressure with a solid sense for setting priorities
• Ability to lead technical work with strong sense of ownership
• Strong command of English language (both verbal and written)