Lead Data Engineer

Talentien; Coimbatore District; Hybrid; INR 2,800,000 - 5,000,000; Full time

Job summary

Talentien is seeking a Lead Data Engineer to design and manage large-scale data pipelines using Spark, Scala, and AWS services. You will drive data modeling, ETL development, and collaboration with data scientists, software engineers, and product teams.

You will mentor peers, ensure high standards for data operations, and help expand real-time and offline analytics capabilities with Tableau dashboards and self-service data sets.

Qualifications

• Bachelor’s degree in computer science, mathematics, statistics or a related quantitative field.
• Experience with Redshift or another columnar store DW
• Experience with cloud solutions / AWS
• Experience building reports and/or data visualization
• Experience with Hadoop/MapReduce/AWS/EMR
• Experience with Spark/Scala/PySpark/ Apache Kafka / Kinesis Streams / AWS
• Glue
• Experience building or administering reporting/analytics platforms
• Experience building flexible data APIs that consumers use to power other parts of the data stack
• 3 to 5 years of experience as a data/software developer/scientist or related technical job
• Experience with predictive analytics/decision models/data mining libraries
• Experience in algorithm design and problem solving
• Experience leading small teams of engineers
• Experience with Agile Development
• Experience with Machine Learning (Classification, Collaborative Filtering)
• Experience with Database development, data modelling and ETL pipeline development using tools like Informatica/Snowflake.

Responsibilities

• Design and analyze big data pipelines and data models.
• Develop and deploy data pipelines using Spark, Scala and AWS EMR.
• Collaborate with Product, BI, ML teams to build datasets and reporting.
• Mentor peers in the Data Analytics team and drive standards for ETL operations.

Skills

pyspark

agile

scala

aws

data

big data

Education

Bachelor’s degree in computer science or related field

Tools

Redshift

S3

EMR

Kinesis

Tableau

Job description

LEAD DATA ENGINEER

Years of Experience – 7 to 12 years

Location – Chennai, Coimbatore & Bangalore

Description

• If you love the challenges that come with big data, then this role is for you. We are looking for Data Engineers who can handle of events a day, manage petabyte scale data on Redshift and S3, and develop data pipelines using Spark/Scala EMR, SQL based ETL, and Java services.
• You are a talented, enthusiastic, and detail-oriented Data Engineer, Data Science, Business Intelligence, or Software Development who knows how to take on big data challenges in an agile way.
• Duties include big data design and analysis data modeling, and development, deployment, and operations of big data pipelines. You will also help hire, mentor, and develop peers in the Data Analytics team including Data Scientists, Data Engineers, and Software Engineers. You'll help Coda build data pipelines and data sets and expand self-service data knowledge.
• This role requires you to live at the cross section of data and engineering. You have a deep understanding of data, analytical techniques, and how to connect insights to the business, and you have practical experience in insisting on highest standards on operations in ETL and big data pipelines.
• We deal in AWS technologies like Redshift, S3, EMR, EC2, DynamoDB, Kinesis Firehose, and Lambda. You'll build our data portfolio and partner with Product, Marketing, BI, and ML teams to build new behavioral events, pipelines, datasets, models, and reporting to support their initiatives.
• You'll also continue to develop our offline analytics capabilities in Tableau and build out our real time dashboarding capabilities.

Basic Qualifications

• Bachelor’s degree in computer science, mathematics, statistics or a similar quantitative field
• Experience with Redshift or another columnar store DW
• Experience with cloud solutions / AWS / Azure / Google cloud
• Experience building reports and/or data visualization
• Experience with Hadoop/MapReduce/AWS/EMR
• Experience with Spark/Scala/PySpark/ Apache Kafka / Kinesis Streams / AWS
• Glue
• Experience building or administering reporting/analytics platforms
• Experience building flexible data APIs that consumers use to power other parts of

The Business Preferred Qualifications

• 3 to 5 years of experience as a data/software developer/scientist or related technical job
• Experience working with predictive analytics/decision models/data mining libraries as well as the tools for developing it
• Experience in algorithm design and problem solving
• Experience leading small teams of engineers
• Experience with Agile Development
• Experience with Machine Learning (Classification, Collaborative Filtering)Experience with Database development, data modelling and ETL pipeline development using tools like Informatica/Snowflake.

Skills: pyspark,agile,scala,aws,data,big data