Lead Data Engineer Spark/Python

Unicon Connectors; Ahmedabad District; On-site; INR 2,500,000 - 4,000,000; Full time

Job summary

Unicon Connectors in India seeks a Lead Data Engineer to spearhead a backend distributed data systems team, shaping the Big Data architecture and data pipelines across Hadoop and NoSQL stacks. You will write and tune Java, MapReduce, Pig and Hive jobs, contribute to open source Big Data technologies, and lead efforts to deploy, audit and support scalable data infrastructure.

You will manage the BI team and vendor partners, own major deliverables, ensure data integrity from source to dashboards,

Qualifications

• Bachelor's or Master's degree in Computer Science, Engineering, or related fields.
• Experience leading a Backend/Distributed Data Systems team is highly desirable.

Responsibilities

• Provide technical leadership in Big Data space and contribute to open source technologies.
• Write and tune complex Java, MapReduce, Pig and Hive jobs.
• Lead the Backend/Distributed Data Systems team while remaining hands-on.
• Lead data infrastructure build, implementation, and support.
• Manage BI team and vendor partners, prioritizing projects and developing dashboards.
• Manage data engineers and deliverables for the Big Data team.

Skills

Python

Spark

Java

Hive

SQL

Hadoop architecture

NoSQL

Scala

MapReduce

C#/.NET

Education

Bachelor's/Master's in Computer Science or related field

Tools

Hadoop

MapReduce

Pig

Hive

Cassandra

HBase

Job description

Responsibilities for this position include :

• Provides technical leadership in Big Data space (Hadoop Stack like M/R, HDFS, Pig, Hive, HBase, Flume, Sqoop, NoSQL stores like Cassandra, HBase etc) across Fractal and contributes to open source Big Data technologies
• Write and tune complex Java, MapReduce, Pig and Hive jobs
• Adapt quickly to change in requirements and be willing to work with different technologies if required
• Experience leading a Backend/Distributed Data Systems team while remaining hands-on is very important
• Lead the effort to build, implement and support the data infrastructure
• Manage the business intelligence team and vendor partners, ensuring to prioritize projects according to customer and internal needs, and develops top-quality dashboards using industry best practices
• Manage team of data engineers (both full-time associates and/or third party resources)
• Own the majority of deliverables for the Big Data team from a delivery perspective
• Analyzes and confirms the integrity of source data to be evaluated
• Leads in deployment and auditing models and attributes for accuracy

Education for Lead Data Engineer :

• Have a relevant degree such as Bachelor's and Master's Degree in Computer Science, Engineering, Statistics, Education, Technical, Information Technology, Information Systems, Mathematics, Computer Engineering, Management

Skills for Lead Data Engineer :

• Python
• Spark
• Java
• Hive
• SQL
• Hadoop architecture
• Large scale search applications and building high volume data pipelines
• Message queuing
• NoSQL
• Scala

Desired experience for lead data engineer includes :

• Experience in development utilizing C# .NET 4.5
• Experience in managing a live service that customers depend
• Experience in coaching and managing other engineers
• Be a team player and enjoy collaboration with other engineers and teams
• Experience with software version management systems
• Experience with task/bug tracking software