Data Engineer - Big Data

micro1; India; Remote; INR 1,500,000 - 2,500,000; Full time

Job summary

A dynamic technology firm is seeking an experienced Data Engineer specializing in Big Data technologies. You will lead a team to design and implement scalable data systems for a global client in the banking sector. The ideal candidate has over 7 years of experience with tools like Hadoop and Spark, and excels in remote collaboration. This role is perfect for those looking to make a significant impact within a thriving team.

Qualifications

• 7-10+ years of experience as a Data Engineer specializing in Big Data.
• Expert in Hadoop and Spark technologies.
• Advanced programming skills in Java and scripting languages.

Responsibilities

• Lead a team of data engineers in delivering Big Data solutions.
• Design and optimize scalable data pipelines.
• Develop high-performance Java and Spark jobs.

Skills

Big Data technologies

Hadoop

Spark

Java

Python

SQL

NoSQL databases

Tools

Hadoop Ecosystem

Azure Cloud

Job description

Overview

Job Type: Full-Time, Contract (6 Months, Renewable)

Location: Remote (Offshore – India)

Summary

Are you passionate about building and optimizing large-scale data platforms? Join our team as a Data Engineer – Big Data, where you will lead cutting-edge data engineering initiatives for a global banking and financial services client. As a technical leader, you will have the opportunity to design, implement, and optimize distributed data systems in a collaborative, asynchronous remote environment.

Responsibilities

• Lead, mentor, and guide our team of data engineers, driving delivery of end-to-end Big Data solutions.
• Design, develop, and optimize scalable data pipelines using Hadoop Ecosystem tools (HDFS, MapReduce, Hive, Pig, HBase, Flume, Sqoop).
• Build, tune, and maintain high-performance Java, MapReduce, Hive, and Spark jobs across the Cloudera Hadoop platform.
• Own the development of dimensional data models and oversee the deployment and auditing of data models for accuracy and quality.
• Develop solutions utilizing NoSQL databases (Cassandra, HBase) and stream-processing frameworks (Spark-Streaming).
• Collaborate with business intelligence teams and vendor partners to prioritize initiatives and deliver robust, business-driven dashboards.
• Champion written communication and asynchronous collaboration to ensure smooth project execution in a global, distributed team.

Required Qualifications

• 7-10+ years of proven experience as a Data Engineer specializing in Big Data technologies.
• Expert-level proficiency in Hadoop (preferably Cloudera), Spark, Pig, Hive, and Impala.
• Advanced programming skills in Java and scripting languages such as Python or Scala.
• In-depth knowledge of database technologies: relational SQL (Postgres) and NoSQL (Cassandra, HBase).
• Track record of designing and implementing scalable dimensional data models and stream-processing systems.
• Exceptional written communication skills, with a strong preference for async collaboration.
• Experience managing teams of engineers and third-party vendors in a remote, offshore setting.

Preferred Qualifications

• Hands-on experience with Azure Cloud services and deployments.
• Background in the Banking or Financial Services industry.
• Familiarity with data science, machine learning tools, and additional scripting languages (C++, Linux, Ruby, PHP, R).