Data Engineer

Blackbuck Insights, LLC; Bengaluru; On-site; INR 1,200,000 - 1,800,000; Full time

Job summary

Blackbuck Insights, LLC is seeking a Data Engineer to design, build, and optimize scalable data pipelines for enterprise analytics, reporting, and AI/ML initiatives.

This role collaborates with architects, analysts, and business stakeholders to deliver reliable data assets in a modern cloud environment. You will implement ETL/ELT processes, ensure data quality and governance, and contribute to Meridian platform integrations and data observability.

Qualifications

• Bachelor's degree in Computer Science, Engineering, Information Systems, or related.
• 5+ years in Data Engineering.
• Strong experience designing enterprise data pipelines and integration.
• Expertise in SQL and Python/Scala/Java.
• Experience with ETL/ELT frameworks and data warehousing.
• Cloud experience (Azure/AWS/GCP) and orchestration tools.
• Knowledge of data quality, governance, and monitoring.

Responsibilities

• Design, develop, and maintain scalable data pipelines.
• Build ETL/ELT processes for ingestion and delivery.
• Develop reusable data engineering components.
• Implement data integration for structured and unstructured data.
• Support cloud-based data platform initiatives.
• Monitor and optimize pipeline performance.
• Collaborate with architects, analysts, and stakeholders.
• Support Meridian platform implementation and enhancements.

Skills

SQL

Python

Scala

Java

ETL/ELT

Data Warehousing

Airflow

Azure Data Factory

Cloud Platforms

AWS

GCP

Databricks

Education

Bachelor's degree in CS/Engineering/IS

Tools

Meridian

Databricks

Snowflake

Azure Synapse

Microsoft Fabric

Kafka

Spark Streaming

Event Hubs

Airflow

CI/CD

IaC

Job description

BBI is a global data engineering consulting firm that empowers clients to effectively scale and modernize. We combine engineering fundamentals and innovative tools to execute business-critical, end-to-end projects on-time and on-budget. We offer expert services across Data Integration, Data Modernization, Data Migration, Data Architecture, Platform Support, and Application Services. Our goal is to provide business value in the most effective way for our clients so clients can focus on growth.

Job Description

Position Summary

The Data Engineer is responsible for designing, building, and optimizing scalable data pipelines, data integration solutions, and cloud-based data platforms that support enterprise analytics, reporting, and AI/ML initiatives.

This role serves as a technical expert in data engineering, ensuring reliable data ingestion, high-quality data assets, and efficient data processing while collaborating closely with architects, analysts, and business stakeholders.

Key Responsibilities

• Design, develop, and maintain scalable and high-performance data pipelines.
  
• Build and optimize ETL/ELT processes to ingest, transform, and deliver data across enterprise platforms.
  
• Develop and maintain reusable data engineering frameworks and components.
  
• Implement data integration solutions for structured and unstructured data sources.
  
• Support modern cloud-based data platform initiatives.
  
• Data Ingestion & Pipeline Operations Develop and manage batch and real-time data ingestion pipelines.
  
• Monitor and optimize pipeline performance, reliability, and scalability.
  
• Troubleshoot data processing issues and implement corrective actions.
  
• Automate data workflows and orchestration processes.
  
• Ensure timely and accurate data delivery to downstream consumers.
  

Data Quality & Governance

• Implement data validation, reconciliation, and quality controls.
  
• Support data governance standards, metadata management, and data lineage initiatives.
  
• Identify and resolve data quality issues and causes.
  
• Contribute to data observability and monitoring solutions.
  
• Ensure compliance with organizational data management standards.
  
• Meridian Platform Support & Implementation Support the implementation and enhancement of Meridian data solutions.
  
• Develop integrations and data pipelines supporting Meridian initiatives.
  
• Collaborate with stakeholders to translate business requirements into technical solutions.
  
• Ensure successful deployment, testing, and operational support of Meridian-related capabilities.
  

Meridian Platform Support & Implementation

• Meridian Platform Support & Implementation Support the implementation and enhancement of Meridian data solutions.
  
• Develop integrations and data pipelines supporting Meridian initiatives.
  
• Collaborate with stakeholders to translate business requirements into technical solutions.
  
• Ensure successful deployment, testing, and operational support of Meridian-related capabilities.
  

Technical Leadership & Collaboration

• Provide technical guidance and mentorship to junior and mid-level engineers.
  
• Participate in architecture reviews, design sessions, and code reviews.
  
• Collaborate with data architects, product owners, analysts, and business stakeholders.
  
• Promote engineering best practices, automation, and continuous improvement.
  
• Document technical designs, standards, and operational procedures.
  

Required Qualifications

• Bachelor's degree in Computer Science, Engineering, Information Systems, or a related field.
  
• 5+ years of experience in Data Engineering or related disciplines.
  
• Strong experience designing and developing enterprise data pipelines and integration solutions.
  
• Expertise in SQL and one or more programming languages such as Python, Scala, or Java.
  
• Experience with ETL/ELT frameworks, data warehousing, and data lake architectures.
  
• Experience with cloud platforms such as Azure, AWS, or Google Cloud. Knowledge of workflow orchestration tools such as Airflow, Azure Data Factory (ADF), or similar platforms.
  
• Strong understanding of data quality, data governance, and monitoring concepts.
  
• Experience troubleshooting and optimizing large-scale data processing solutions.
  

Requirements

• Experience with Meridian platform implementation or support.
  
• Experience with Data bricks, Snowflake, Azure Synapse, Microsoft Fabric, or similar modern data platforms.
  
• Knowledge of streaming technologies such as Kafka, Spark Streaming, or Event Hubs.
  
• Familiarity with DevOps practices, CI/CD pipelines, Infrastructure as Code (IaC), and automated testing.
  
• Experience supporting AI/ML data pipelines and feature engineering.
  
• Relevant cloud, data engineering, or platform certifications.
  
• Success Measures Reliable and scalable data pipelines with minimal operational issues.
  
• Improved data quality and reduced production defects. Timely delivery of data engineering solutions and enhancements.
  
• Successful implementation and support of Meridian data initiatives.
  
• Increased automation, operational efficiency, and platform stability.
  
• High-quality technical documentation and adherence to engineering standards.