Associate Technical Lead | Data Engineer

TO THE NEW; Dadri; On-site; INR 1,200,000 - 2,000,000; Full time

Job summary

TO THE NEW in India is seeking a Senior Data Engineer to design and develop data applications, ensuring timely delivery and high quality. You will collaborate with software teams to implement best practices and optimize performance across big data platforms.

The role requires 6+ years in data engineering with Python/Scala/Java, Hadoop ecosystem, Spark, SQL and NoSQL technologies. You will translate business challenges into scalable solutions and stay current with emerging big data trends.

Qualifications

• BE/B.Tech/MCA with at least 6+ years of experience in data engineering design and development.
• Mandatory experience in following areas: Python/Scala/Java, Hadoop/HDFS/MR, Spark SQL/Dataframes/RDDs, SQL, Hive/Snowflake/SQL Server/Databricks, Elastic Search.
• Preferred experience in 3+ of: Spark Streaming/Spark ML, Kafka/Flume, Apache NiFi, Airflow/Oozie, Cloud-based Data Platforms, NoSQL databases.
• Good knowledge of current tech landscape and ability to visualize industry trends.
• Working knowledge of Big Data integration with metadata management, data quality, and MDM.

Responsibilities

• Review requirements and ensure development tasks meet timelines and are thoroughly tested.
• Partner with development teams to implement best practices and optimize data applications.
• Collaborate with customers to translate business challenges into innovative solutions.
• Research new Big Data technologies and assess alignment with strategy.
• Work in an agile process to increase speed to market while maintaining controls.

Skills

Python/Scala/Java

Hadoop/HDFS/MR

Spark SQL/Dataframes/RDDs

SQL

Hive/Snowflake/SQL Server/Databricks

Elastic Search

Education

BE/B.Tech/MCA

Job description

Role & responsibilities:

• Review and understand business requirements ensuring that development tasks are completed within the timeline provided and that issues are fully tested with minimal defects
• Partner with a software development team to implement best practices and optimize the performance of Data applications to ensure that client needs are met at all times
• Collaborate across the company and interact with our customers to understand, translate, define, design their business challenges and concerns into innovative solutions
• Research on new Big Data technologies, assessing maturity and alignment of technology to business and technology strategy
• Work in a rapid and agile development process to enable increased speed to market while maintaining appropriate controls

Preferred candidate profile:

• BE/B.Tech/MCA with At least 6+ years of experience in design and development using Data Engineering technology stack and programming languages
• Mandatory experience in following areas:

1. Python/Scala/Java
2. Hadoop, HDFS, MR
3. Spark SQL, Dataframes, RDDs
4. SQL
5. Hive / Snowflake/SQL Server/Databricks
6. Elastic Search

• Preferred experience in 3 or more of the following areas:

1. Spark Streaming, Spark ML
2. Kafka/Flume
3. Apache NiFi
4. Apache Airflow/Oozie
5. Cloud-based Data Platforms
6. NoSQL Databases HBase/Cassandra/Neo4j/MongoDB

• Good knowledge of the current technology landscape and ability to visualize industry trends
• Working knowledge of Big Data Integration with Third-party or in-house built Metadata Management, Data Quality, and Master Data Management solutions
• Active community involvement through articles,blogs or speaking engagements at conferences