Lead Data Engineer - Spark/Python

Unicon Connectors; Pune District; On-site; INR 1,500,000 - 2,100,000; Full time

Benefits offered by this job

Job summary

Unicon Connectors in Pune district, India, seeks a Lead Data Engineer to shape and deliver our Big Data platform, guiding a backend/distributed data systems team while remaining hands-on with core technologies. You will own the data architecture, build scalable data pipelines with Python, Spark, Java and Hadoop, and mentor engineers, while delivering high-quality dashboards and analytics for customers.

The role requires collaboration with cross-functional teams and vendors, and a focus on data

Qualifications

• Bachelor's or Master's degree in CS, IT, or related field.
• Experience leading a Backend/Distributed Data Systems team.
• Hands-on with Big Data technologies and data infrastructure.
• Ability to adapt to changing requirements and new tech.
• Experience with live services and coaching engineers.
• Familiarity with BI dashboards and data governance.

Responsibilities

• Provide technical leadership in Big Data tech stack across Fractal.
• Write and tune complex Java, MapReduce, Pig and Hive jobs.
• Lead data infra builds, deployment, and auditing models.
• Manage BI team and vendor partners; develop dashboards.
• Manage data engineering team and deliverables.
• Analyze source data integrity and verify accuracy.
• Oversee deployment and monitoring of data pipelines.

Skills

Python

Spark

Java

Hive

SQL

Hadoop architecture

NoSQL

Scala

Message queuing

Education

Bachelor's degree in Computer Science or related field

Master's degree in Computer Science or related field

Job description

Responsibilities

• Provides technical leadership in Big Data space (Hadoop Stack like M/R, HDFS, Pig, Hive, HBase, Flume, Sqoop, NoSQL stores like Cassandra, HBase etc) across Fractal and contributes to open source Big Data technologies
• Write and tune complex Java, MapReduce, Pig and Hive jobs
• Adapt quickly to change in requirements and be willing to work with different technologies if required
• Experience leading a Backend/Distributed Data Systems team while remaining hands-on is very important
• Lead the effort to build, implement and support the data infrastructure
• Manage the business intelligence team and vendor partners, ensuring to prioritize projects according to customer and internal needs, and develops top-quality dashboards using industry best practices
• Manage team of data engineers (both full-time associates and/or third party resources)
• Own the majority of deliverables for the Big Data team from a delivery perspective
• Analyzes and confirms the integrity of source data to be evaluated
• Leads in deployment and auditing models and attributes for accuracy

Education for Lead Data Engineer

Have a relevant degree such as Bachelor's and Master's Degree in Computer Science, Engineering, Statistics, Education, Technical, Information Technology, Information Systems, Mathematics, Computer Engineering, Management

Skills for Lead Data Engineer

• Desired skills for lead data engineer include :
• Python
• Spark
• Java
• Hive
• SQL
• Hadoop architecture
• Large scale search applications and building high volume data pipelines
• Message queuing
• NoSQL
• Scala

Desired experience for lead data engineer includes

• Experience in development utilizing C# .NET 4.5
• Experience in managing a live service that customers depend
• Experience in coaching and managing other engineers
• Be a team player and enjoy collaboration with other engineers and teams
• Experience with software version management systems
• Experience with task/bug tracking software