Data Engineer

Blend

Hyderabad

On-site

INR 1,000,000 - 1,500,000

Full time

14 days+

Get more replies from employers

Send a job-specific resume in minutes.

Job summary

A leading data and AI services company in Hyderabad is seeking a Data Engineer to build and maintain resilient data pipelines for a Media Mix Optimization platform. The ideal candidate will have robust experience with HDFS, Apache NiFi, and SQL. This role emphasizes data quality and governance within a collaborative environment. The position offers opportunities for growth in a dynamic setting.

Qualifications

Proven experience as a Data Engineer with strong background in ETL/ELT design.
Experience with on-premises big data ecosystems supporting distributed computing.
Ability to work in agile environments, collaborating with multi-disciplinary teams.

Responsibilities

Design, build, and optimize scalable data pipelines in Apache NiFi.
Architect and manage datasets on HDFS for high-volume storage.
Maintain and tune Postgres databases for high availability and performance.
Implement monitoring, logging, and alerting frameworks for data pipelines.
Enforce standards for data quality, lineage, and security across systems.

Skills

HDFS

Apache NiFi

Hive

PySpark

Postgres

Python

SQL

ETL/ELT design

Distributed processing

Education

Bachelor’s degree in Computer Science, Information Technology, or related field

Master’s degree preferred

Tools

Airflow

Oozie

Job description

Overview

Join to apply for the Data Engineer role at Blend

Company Description

Blend360 is a data and AI services company specializing in data engineering, data science, MLOps, and governance to build scalable analytics solutions. It partners with enterprise and Fortune 1000 clients across industries including financial services, healthcare, retail, technology, and hospitality to drive data-driven decision making. Headquartered in Columbia, Maryland, the company is recognized for rapid growth and global delivery of AI solutions through the integration of people, data, and technology.

Job Description

We are implementing a Media Mix Optimization (MMO) platform designed to analyze and optimize marketing investments across multiple channels. This initiative requires a robust on-premises data infrastructure to support distributed computing, large-scale data ingestion, and advanced analytics. The Data Engineer will be responsible for building and maintaining resilient pipelines and data systems that feed into MMO models, ensuring data quality, governance, and availability for Data Science and BI teams. The environment integrates HDFS for distributed storage, Apache NiFi for orchestration, Hive and PySpark for distributed processing, and Postgres for structured data management. This role is central to enabling seamless integration of massive datasets from disparate sources (media, campaign, transaction, customer interaction, etc.), standardizing data, and providing reliable foundations for advanced econometric modeling and insights.

Responsibilities

Data Pipeline Development & Orchestration: Design, build, and optimize scalable data pipelines in Apache NiFi to automate ingestion, cleansing, and enrichment from structured, semi-structured, and unstructured sources. Ensure pipelines meet low-latency and high-throughput requirements for distributed processing.
Data Storage & Processing: Architect and manage datasets on HDFS to support high-volume, fault-tolerant storage. Develop distributed processing workflows in PySpark and Hive to handle large-scale transformations, aggregations, and joins across petabyte-level datasets. Implement partitioning, bucketing, and indexing strategies to optimize query performance.
Database Engineering & Management: Maintain and tune Postgres databases for high availability, integrity, and performance. Write advanced SQL queries for ETL, analysis, and integration with downstream BI/analytics systems.
Collaboration & Integration: Partner with Data Scientists to deliver clean, reliable datasets for model training and MMO analysis. Work with BI engineers to ensure data pipelines align with reporting and visualization requirements.
Monitoring & Reliability Engineering: Implement monitoring, logging, and alerting frameworks to track data pipeline health. Troubleshoot and resolve issues in ingestion, transformations, and distributed jobs.
Data Governance & Compliance: Enforce standards for data quality, lineage, and security across systems. Ensure compliance with internal governance and external regulations.
Documentation & Knowledge Transfer: Develop and maintain comprehensive technical documentation for pipelines, data models, and workflows. Provide knowledge sharing and onboarding support for cross-functional teams.

Qualifications

Bachelor’s degree in Computer Science, Information Technology, or related field (Master’s preferred).
Proven experience as a Data Engineer with expertise in HDFS, Apache NiFi, Hive, PySpark, Postgres, Python, and SQL.
Strong background in ETL/ELT design, distributed processing, and relational database management.
Experience with on-premises big data ecosystems supporting distributed computing.
Solid debugging, optimization, and performance tuning skills.
Ability to work in agile environments, collaborating with multi-disciplinary teams.
Strong communication skills for cross-functional technical discussions.

Preferred Qualifications

Familiarity with data governance frameworks, lineage tracking, and data cataloging tools.
Knowledge of security standards, encryption, and access control in on-premises environments.
Prior experience with Media Mix Modeling (MMM/MMO) or marketing analytics projects.
Exposure to workflow schedulers (Airflow, Oozie, or similar).
Proficiency in developing automation scripts and frameworks in Python for CI/CD of data pipelines.

Seniority level