Data Engineering Lead | Hybrid | To THE New

TO THE NEW; Sector 10; Hybrid; INR 2,800,000 - 6,000,000; Full time

Job summary

TO THE NEW is seeking a seasoned Data Engineer with strong Python/Scala/Java skills and a deep Hadoop/Spark/SQL background.

The role focuses on optimizing data applications, designing scalable data pipelines, and mentoring junior engineers in an agile environment.

Qualifications

• BE/B.Tech/MCA required with relevant IT experience.
• 7+ years total IT experience with 4+ years in Hadoop tech stack.
• Hands-on in 2+ areas from the listed Hadoop ecosystem.

Responsibilities

• Review requirements and ensure tasks meet timelines and are thoroughly tested.
• Collaborate with software teams to optimize data applications for client needs.
• Interact with customers to translate business challenges into data solutions.
• Research new Big Data technologies and assess their maturity for business use.
• Work in an agile process to balance speed with control.

Skills

Python/Scala/Java

Data engineering

Team mentoring

Big Data architecture

Education

BE/B.Tech/MCA

Tools

Hadoop

HDFS

MR

Spark

Kafka

Flume

NiFi

Hortonworks

Hive

Pig

Sqoop

NoSQL (HBase/Cassandra/MongoDB)

D3.js

Grafana

Kibana

Tableau

Pentaho

Scrapy

ElasticSearch

Google Analytics

Kerberos/OpenLDAP/Knox/Ranger

Job description

Job Summary:

The ideal candidate would be a skilled Data Engineer proficient in Python, Scala, or Java with a strong background in Hadoop, Spark, SQL, and various data platforms and have expertise in optimizing the performance of data applications and contributing to rapid and agile development processes. Team handling and mentoring experience is a plus.

Responsibilities

• Review and understand business requirements ensuring that development tasks are completed within the timeline provided and that issues are fully tested with minimal defects
• Partner with a software development team to implement best practices and optimize the performance of Data applications to ensure that client needs are met at all times
• Collaborate across the company and interact with our customers to understand, translate, define, and design their business challenges and concerns into innovative solutions
• Research on new Big Data technologies, assessing maturity and alignment of technology to business and technology strategy
• Work in a rapid and agile development process to enable increased speed to market while maintaining appropriate controls

Requirements

• BE/B.Tech/MCA with At least 7+ years of total IT experience with 4+ years of experience in design and development using Hadoop technology stack and programming languages
• Preferred hands-on experience in 2 or more areas:
• Hadoop, HDFS, MR
• Spark Streaming, Spark SQL, Spark ML
• Kafka/Flume
• Apache NiFi
• Worked with Hortonworks Data Platform
• Hive / Pig / Sqoop
• NoSQL Databases HBase/Cassandra/Neo4j/MongoDB
• Visualization & Reporting frameworks like D3.js, Zeppelin, Grafana, Kibana Tableau, Pentaho
• Scrapy for crawling websites
• Elastic Search
• Google Analytics data streaming
• Data security (Kerberos/Open LDAP/Knox/Ranger)
• Good knowledge of the current technology landscape and ability to visualize industry trends
• Working knowledge of Big Data integration with third-party or in-house built Metadata Management, Data Quality, Master Data Management solutions, structured/unstructured data
• Active community involvement in terms of articles, blogs or speaking engagements at conferences