Senior Data Engineer Data Mining & Wrangling

Unicon Connectors; India; On-site; INR 1,200,000 - 2,200,000; Full time

Job summary

Unicon Connectors in India is seeking a data engineering professional to enhance our data lake and pipelines. You will collaborate with engineering, manufacturing, and quality teams to identify data needs and build scalable data solutions.

The role emphasizes data wrangling, transformation, and reporting to drive process improvements across factories. The candidate should excel in unstructured data analytics, big data tools, and cross-functional collaboration, contributing to robust data

Qualifications

• Experience solving manufacturing use cases and improving processes using data.
• Analytical skills for unstructured datasets and data wrangling
• Ability to define data requirements and collect data with connectivity engineers
• Experience building scalable big data pipelines and data stores
• Knowledge of data transformation, metadata, and workload management
• Familiarity with message queues and stream processing tools

Responsibilities

• Understand factories, processes, data availability and improvement avenues.
• Brainstorm with engineering, manufacturing and quality to solve problems via data lake.
• Define required data and coordinate with connectivity engineers to collect it.
• Create and maintain optimal data pipeline architecture.
• Assemble large, complex datasets meeting business requirements.
• Automate processes and optimize data delivery for scalability.
• Prepare data, deep-dive analyses, and help teams understand machine behavior.
• Deploy and monitor solutions.
• Collaborate with data analytics to enhance data systems.
• Work with Data Architects and data modeling teams.

Skills

Manufacturing data use cases

Analytics on unstructured data

Data transformation & pipelines

Data delivery optimization

Data discovery & scoping

Numerical & analytical skills

Tools

Spark

Delta Lake

CDC

NiFi

Kafka

SQL & NoSQL (Oracle, Hive)

Scala

Java

C++

PowerBI/Tableau

Job description

Key Responsibilities

• Understand the factories , manufacturing process , data availability and avenues for improvement
• Brainstorm , together with engineering, manufacturing and quality problems that can be solved using the acquired data in the data lake platform.
• Define what data is required to create a solution and work with connectivity engineers , users to collect the data
• Create and maintain optimal data pipeline architecture.
• Assemble large, complex data sets that meet functional / non-functional business requirements.
• Identify, design, and implement internal process improvements: automating manual processes, optimizing data delivery for greater scalability
• Work on data preparation, data deep dive , help engineering, process and quality to understand the process/ machine behavior more closely using available data
• Deploy and monitor the solution
• Work with data and analytics experts to strive for greater functionality in our data systems.
• Work together with Data Architects and data modeling teams.

Skills /Competencies

• Good knowledge of the business vertical with prior experience in solving different use cases in the manufacturing or similar industry
• Ability to bring cross industry learning to benefit the use cases aimed at improving manufacturing process

Problem Scoping/definition Skills

• Experience in problem scoping, solving, quantification
• Strong analytic skills related to working with unstructured datasets.
• Build processes supporting data transformation, data structures, metadata, dependency and workload management.
• Working knowledge of message queuing, stream processing, and highly scalable 'big data' data stores
• Ability to foresee and identify all right data required to solve the problem

Data Wrangling Skills

• Strong skill in data mining, data wrangling techniques for creating the required analytical dataset
• Experience building and optimizing 'big data' data pipelines, architectures and data sets
• Adaptive mindset to improvise on the data challenges and employ techniques to drive desired outcomes

Programming Skills

• Experience with big data tools: Spark, Delta, CDC, NiFi, Kafka, etc
• Experience with relational SQL ,NoSQL databases and query languages, including oracle , Hive, sparkQL.
• Experience with object-oriented languages: Scala, Java, C++ etc.

Visualization Skills

• Know how of any visualization tools such as PowerBI, Tableau
• Good storytelling skills to present the data in simple and meaningful manner

Data Engineering Skills

• Strong skill in data analysis techniques to generate finding and insights by means of exploratory data analysis
• Good understanding of how to transform and connect the data of various types and form
• Great numerical and analytical skills
• Identify opportunities for data acquisition
• Explore ways to enhance data quality and reliability
• Build algorithms and prototypes
• Reformulating existing frameworks to optimize their functioning.
• Good understanding of optimization techniques to make the system performant for requirements.