Lead Data Engineer - Spark / Python

Unicon Connectors; Chennai District; On-site; INR 2,800,000 - 5,200,000; Full time

Job summary

Unicon Connectors in Chennai is seeking a Lead Data Engineer to drive our Big Data initiatives. You will provide technical leadership across the Hadoop stack, and contribute to scalable data infrastructure while staying hands-on.

You will manage a data engineering team and vendor partners, prioritise projects to customer needs, and deliver high quality dashboards using industry best practices. A relevant bachelor's or master's degree is required and 5–7 years of experience in backend/distributed

Qualifications

• Bachelor's or Master's degree in Computer Science, Engineering, or a related field.
• 5–7 years of experience in backend/distributed data systems.
• Experience leading a data engineering team and delivering complex Big Data projects.

Responsibilities

• Provide technical leadership in Big Data space across Hadoop stack and related technologies.
• Write and tune complex Java, MapReduce, Pig and Hive jobs; adapt to changing requirements.
• Lead a Backend/Distributed Data Systems team while remaining hands-on and deliver data infrastructure.
• Manage BI team and vendor partners; prioritize projects to customer/internal needs and develop dashboards.
• Own deliverables for the Big Data team from a delivery perspective; verify data integrity and lead deployment and auditing models.

Skills

Python

Spark

Java

Hive

SQL

Hadoop architecture

Large scale data pipelines

Message queuing

NoSQL

Scala

C# .NET

Education

Bachelor's or Master's in CS/Engineering

Job description

Responsibilities for this position include :

• Provides technical leadership in Big Data space (Hadoop Stack like M/R, HDFS, Pig, Hive, HBase, Flume, Sqoop, NoSQL stores like Cassandra, HBase etc) across Fractal and contributes to open source Big Data technologies
• Write and tune complex Java, MapReduce, Pig and Hive jobs- Adapt quickly to change in requirements and be willing to work with different technologies if required
• Experience leading a Backend/Distributed Data Systems team while remaining hands-on is very important- Lead the effort to build, implement and support the data infrastructure
• Manage the business intelligence team and vendor partners, ensuring to prioritize projects according to customer and internal needs, and develops top-quality dashboards using industry best practices- Manage team of data engineers (both full-time associates and/or third party resources)
• Own the majority of deliverables for the Big Data team from a delivery perspective- Analyzes and confirms the integrity of source data to be evaluated- Leads in deployment and auditing models and attributes for accuracy

Education for Lead Data Engineer :Have a relevant degree such as Bachelor's and Master's Degree in Computer Science, Engineering, Statistics, Education, Technical, Information Technology, Information Systems, Mathematics, Computer Engineering, Management Skills for Lead Data Engineer

Desired skills for lead data engineer include :- Python- Spark- Java- Hive- SQL- Hadoop architecture- Large scale search applications and building high volume data pipelines- Message queuing- NoSQL

• Python
• Spark
• Java
• Hive
• SQL
• Hadoop architecture
• Large scale search applications and building high volume data pipelines
• Message queuing
• NoSQL

- ScalaDesired experience for lead data engineer includes :- Experience in development utilizing C# .NET 4.5

• Scala
• Experience in development utilizing C# .NET 4.5

• Experience in managing a live service that customers depend
• Experience in coaching and managing other engineers
• Be a team player and enjoy collaboration with other engineers and teams
• Experience with software version management systems
• Experience with task/bug tracking software

Exp : 5-7yrs