Associate Technical Lead - Data Engineering

To The New; Dadri; On-site; INR 1,800,000 - 2,800,000; Full time

Job summary

TO THE NEW is seeking an Associate Technical Lead – Data Engineering to guide Hadoop-based data platforms and collaborate with clients on scalable data solutions.

You will lead design and development, drive best practices, and explore Spark, Kafka, and related technologies to deliver robust data applications in an agile environment.

Qualifications

• BE/B.Tech/MCA with 6+ years total IT experience and 4+ years in design and development using Hadoop stack.
• Hands-on in 2+ areas: Hadoop ecosystem, Spark, Kafka/Flume, or related big data tools.
• Strong knowledge of data security, metadata management and data quality concepts.

Responsibilities

• Review requirements and ensure development tasks meet timelines and are thoroughly tested with minimal defects.
• Partner with the software team to implement best practices and optimize data application performance.
• Collaborate with customers to translate business challenges into innovative data solutions.
• Research new big data technologies and assess fit with business strategy.
• Contribute in agile processes to balance speed with controls.

Skills

Hadoop

HDFS

MR

Spark Streaming

Spark SQL

Spark ML

Kafka

Flume

Hortonworks

Scrapy

Elasticsearch

Data security

Kerberos

OpenLDAP

Knox

Ranger

Metadata Mgmt

Data Quality

MDM

Education

BE/B.Tech/MCA

Tools

Scrapy

Hortonworks

Kafka

Elasticsearch

Job description

Associate Technical Lead – Data Engineering

• Review and understand business requirements ensuring that development tasks are completed within the timeline provided and that issues are fully tested with minimal defects
• Partner with a software development team to implement best practices and optimize the performance of Data applications to ensure that client needs are met at all times
• Collaborate across the company and interact with our customers to understand, translate, define, and design their business challenges and concerns into innovative solutions
• Research on new Big Data technologies, assessing maturity and alignment of technology to business and technology strategy
• Work in a rapid and agile development process to enable increased speed to market while maintaining appropriate controls

Requirements

• BE/B.Tech/MCA with At least 6+ years of total IT experience with 4+ years of experience in design and development using Hadoop technology stack and programming languages
• Preferred hands-on experience in 2 or more areas:
• Hadoop, HDFS, MR
• Spark Streaming, Spark SQL, Spark ML
• Kafka/Flume
• Worked with Hortonworks Data Platform
• Scrapy for crawling websites
• Elastic Search
• Data security (Kerberos/Open LDAP/Knox/Ranger)
• Good knowledge of the current technology landscape and ability to visualize industry trends
• Working knowledge of Big Data integration with third-party or in-house built Metadata Management, Data Quality, Master Data Management solutions, structured/unstructured data
• Active community involvement in terms of articles, blogs or speaking engagements at conferences

Life is Better Together

A glimpse into the milestones, wins, and everyday moments that shape life at TO THE NEW.

Celebrating milestones. Inspiring what's next

Geek Combat (Hackathon)

Turning bold ideas into real solutions

Marathon

Going the extra mile, together

Diwali celebrations

Bringing festive spirit to work

Empowering every voice, every day

Bootcamp

Learning together. Growing every day

Ready to write your next chapter?

Join a team that values your voice as much as your code. We're hiring across all departments.