Principal Data Engineer

Nxtwave Disruptive Technologies (Hiring for a Client); Bangalore Rural, Bengaluru; On-site; INR 4,500,000 - 7,500,000; Full time

Job summary

Nxtwave Disruptive Technologies (Hiring for a Client) seeks a Principal Data Engineer with 10-12 years to design, build, and scale a fintech data platform. You will own architectural strategy for high-performance data lakes, real-time pipelines, and scalable warehouse models, collaborating with cross-functional stakeholders.

You will write complex SQL, lead streaming infrastructure with Spark/Flink, and drive DevOps practices, CI/CD, and data governance in a demanding fintech environment.

Qualifications

• 10-12 years of direct, hands-on software development experience within a dedicated Data Engineer or Data Architect role.
• Comprehensive expertise utilizing big data infrastructure ecosystems: HDFS, S3, Spark, Flink, Hive, HBase, Kafka, or Kinesis.
• Advanced proficiency in relational SQL querying along with NoSQL databases (Elasticsearch, Cassandra, MongoDB).
• Solid command of workflow orchestration tools (Airflow, Luigi, Azkaban) and programming languages (Java, Scala, Python).
• Prior fintech or high-volume transactional platform experience is highly preferred.

Responsibilities

• Pipeline Architecture: Design, build, and maintain real-time and batch data pipelines using modern Big Data technologies.
• Warehousing & Lakes: Develop high-performance data warehouses and data lakes on AWS/GCP with safe writes and scalable schemas.
• Data Modeling & Taxonomy: Create logical/conceptual data models and a centralized metadata taxonomy for stakeholders.
• Stream Processing: Implement real-time processing within a microservices architecture using Spark-Streaming or Flink.
• Process Automation & DevOps: Promote CI/CD, TDD, automated tracking, and performance optimization across the team.
• API Integrations: Build data ingestion frameworks for secure external datasets via APIs and WebSockets.

Skills

Data architecture

Big data engineering

SQL proficiency

Cloud platforms

Leadership & collaboration

Tools

HDFS

S3

Spark

Flink

Hive

HBase

Kafka

Kinesis

Elasticsearch

Cassandra

MongoDB

Airflow

Luigi

Azkaban

Java

Scala

Python

Job description

Job Overview:

We are seeking a seasoned Principal Data Engineer with 10-12 years of experience to design, build, and scale our next-generation fintech data platform. This is a senior technical leadership role where you will own the architectural strategy for high-performance data lakes, real-time distributed data pipelines, and scalable warehouse models. The ideal candidate is equally comfortable collaborating with cross-functional business stakeholders to define core taxonomies as they are writing complex SQL and stream-processing infrastructure.

Key Responsibilities:

• Pipeline Architecture: Design, build, and maintain highly optimal real-time and batch data pipeline architectures using modern Big Data technologies.
• Warehousing & Lakes: Develop and maintain robust, high-performance data warehouses and data lakes on AWS/GCP to guarantee safe writes, efficient retrieval, and schema evolution.
• Data Modeling & Taxonomy: Author complex SQL queries, design logical/conceptual data models, and create a centralized metadata taxonomy/data dictionary for business stakeholders.
• Stream Processing: Implement real-time data processing pipelines within a microservices architecture using stream tools like Spark-Streaming or Flink.
• Process Automation & DevOps: Advocate for and implement engineering best practices across the team, driving automated tracking, CI/CD deployment logic, TDD workflows, and performance optimization.
• API Integrations: Build data acquisition frameworks to cleanly ingest external datasets through secure enterprise APIs and WebSockets.

Required Skills & Qualifications:

• 10-12 years of direct, hands-on software development experience within a dedicated Data Engineer or Data Architect role.
• Comprehensive expertise utilizing big data infrastructure ecosystems: HDFS, S3, Spark, Flink, Hive, Hbase, Kafka, or Kinesis.
• Advanced proficiency in relational SQL querying along with NoSQL database management systems (Elasticsearch, Cassandra, MongoDB).
• Solid command over workflow orchestration tools (Airflow, Luigi, Azkaban) and object-oriented/functional programming scripting languages (Java, Scala, Python).
• Prior experience working within a dynamic fintech or high-volume transactional platform environment is highly preferred.