Data Engineer

Vistaar Finance; Bengaluru; Hybrid; INR 1,200,000 - 1,800,000; Full time

Job summary

Vistaar Finance is seeking a Data Engineer specializing in Data Integration to design and implement ETL/ELT processes that move data from operational and external environments into the business intelligence layer. You will build scalable pipelines, ensure data quality, and optimize performance in a data warehouse environment.

The role emphasizes batch and near real-time data processing using AWS Glue and Apache Kafka, with a focus on data modeling, governance, and cost-aware architecture within

Qualifications

• 2+ years of experience in Data Engineering or related roles.
• Proficient in Advanced SQL with performance tuning and window functions.
• Hands-on Python for data processing and scripting.
• Experience building ETL/ELT pipelines using AWS Glue or similar tools.
• Working knowledge of streaming systems such as Apache Kafka.
• Familiarity with AWS services (S3, Redshift, DMS) and data modeling concepts.

Responsibilities

• Design and build scalable data integration pipelines ingesting data from multiple sources into the data warehouse.
• Develop batch and near real-time ETL/ELT pipelines using AWS Glue or Kafka-based frameworks.
• Implement data transformation, cleansing, and validation to ensure data quality.
• Optimize pipelines and queries for performance, scalability, and cost efficiency (Redshift emphasis).
• Monitor, troubleshoot, and resolve data pipeline issues for high availability.
• Collaborate with stakeholders to understand data requirements and deliver robust data solutions.
• Maintain documentation and follow best practices in data engineering, including logging and version control.

Skills

Advanced SQL

Python

Data modeling

ETL/ELT pipelines

Batch & real-time pipelines

Streaming systems

Tools

AWS Glue

Apache Kafka

Amazon DMS

Amazon Redshift

S3

PySpark

Job description

Roles and responsibilities

As a Data Engineer specializing in Data Integration, you will design and build solutions to transfer data from operational and external environments to the business intelligence environment. You will utilize various tools to create and implement Extract, Transform, and Load (ETL) processes, ensuring the seamless flow of data throughout the business intelligence solution's lifecycle. Your primary responsibilities will include:

• Design and develop scalable data integration pipelines to ingest data from multiple sources (core systems, third-party systems, and streaming platforms) into the data warehouse.
• Build and maintain batch and near real-time ETL/ELT pipelines using tools such as AWS Glue, AWS DMS or Apache Kafka.
• Implement data transformation, cleansing, and validation logic to ensure high data quality and consistency.
• Optimize data pipelines and queries for performance, scalability, and cost efficiency, especially within Amazon Redshift.
• Monitor, troubleshoot, and resolve data pipeline issues, ensuring high availability and reliability of data systems.
• Collaborate with stakeholders to understand data requirements and deliver robust data solutions.
• Maintain proper documentation and follow best practices in data engineering, including logging, error handling, and version control.

Required Skills & Qualifications

• 2+ years of experience in Data Engineering or related roles
• Strong proficiency in Advanced SQL (joins, window functions, query optimization, CTE function)
• Hands-on experience with Python for data processing
• Experience building ETL/ELT pipelines using tools like AWS Glue or similar frameworks
• Working knowledge of streaming systems such as Apache Kafka
• Familiarity with AWS services such as Amazon S3, Amazon Redshift, and AWS DMS
• Understanding of data modeling concepts (OLAP vs OLTP, star schema, normalization)
• Experience working with different types of databases, such as Relational (SQL) Databases, NoSQL Databases, Data Warehouses (OLAP) etc with a clear understanding of their use cases, performance characteristics, and data modeling approaches.

Good to Have

• Experience with Apache Spark / PySpark
• Knowledge of data quality frameworks and monitoring tools
• Understanding of cloud cost optimization techniques
• Some exposure in LLM training, hosting, modelling creation of vector database etc.