Lead Data Engineer

Talentien; Chennai District; On-site; INR 3,500,000 - 7,000,000; Full time

Job summary

Talentien is seeking a Lead Data Engineer in Chennai with 7–12 years of experience to own end-to-end data pipelines on AWS. You will design data models, build scalable ETL processes, and partner with cross-functional teams to deliver insights.

The role involves mentoring engineers, advancing offline and real-time analytics, and ensuring high standards in data operations. Expect collaboration with Product, BI, and ML teams across multiple cities.

Qualifications

• Bachelor’s degree in computer science, mathematics, statistics or a similar quantitative field.
• Experience with Redshift or another columnar store DW.
• Experience with cloud solutions / AWS / Azure / Google cloud.
• Experience building reports and/or data visualization.
• Experience with Hadoop/MapReduce/AWS/EMR.
• Experience with Spark/Scala/PySpark/ Apache Kafka / Kinesis Streams / AWS.
• Glue
• Experience building or administering reporting/analytics platforms.
• Experience building flexible data APIs that consumers use to power other parts of

Responsibilities

• Design and analyze big data models and pipelines.
• Develop, deploy, and operate big data pipelines.
• Mentor and develop peers in the Data Analytics team.
• Collaborate with Product, Marketing, BI, and ML teams to build datasets and dashboards.

Skills

pyspark

agile

scala

aws

data

big data

Education

Bachelor's degree in computer science, mathematics, statistics or a similar quantitative field

Tools

Redshift

S3

EMR

Spark

Kafka

Kinesis

Job description

LEAD DATA ENGINEER

Years of Experience – 7 to 12 years

Location – Chennai, Coimbatore & Bangalore

Description

• If you love the challenges that come with big data, then this role is for you. We are looking for Data Engineers who can handle of events a day, manage petabyte scale data on Redshift and S3, and develop data pipelines using Spark/Scala EMR, SQL based ETL, and Java services.
• You are a talented, enthusiastic, and detail-oriented Data Engineer, Data Science, Business Intelligence, or Software Development who knows how to take on big data challenges in an agile way.
• Duties include big data design and analysis data modeling, and development, deployment, and operations of big data pipelines. You will also help hire, mentor, and develop peers in the Data Analytics team including Data Scientists, Data Engineers, and Software Engineers. You'll help Coda build data pipelines and data sets and expand self-service data knowledge.
• This role requires you to live at the cross section of data and engineering. You have a deep understanding of data, analytical techniques, and how to connect insights to the business, and you have practical experience in insisting on highest standards on operations in ETL and big data pipelines.
• We deal in AWS technologies like Redshift, S3, EMR, EC2, DynamoDB, Kinesis Firehose, and Lambda. You'll build our data portfolio and partner with Product, Marketing, BI, and ML teams to build new behavioral events, pipelines, datasets, models, and reporting to support their initiatives.
• You'll also continue to develop our offline analytics capabilities in Tableau and build out our real time dashboarding capabilities.

Basic Qualifications

• Bachelor’s degree in computer science, mathematics, statistics or a similar quantitative field
• Experience with Redshift or another columnar store DW
• Experience with cloud solutions / AWS / Azure / Google cloud
• Experience building reports and/or data visualization
• Experience with Hadoop/MapReduce/AWS/EMR
• Experience with Spark/Scala/PySpark/ Apache Kafka / Kinesis Streams / AWS
• Glue
• Experience building or administering reporting/analytics platforms
• Experience building flexible data APIs that consumers use to power other parts of

The Business Preferred Qualifications

• 3 to 5 years of experience as a data/software developer/scientist or related technical job
• Experience working with predictive analytics/decision models/data mining libraries as well as the tools for developing it
• Experience in algorithm design and problem solving
• Experience leading small teams of engineers
• Experience with Agile Development
• Experience with Machine Learning (Classification, Collaborative Filtering)Experience with Database development, data modelling and ETL pipeline development using tools like Informatica/Snowflake.

Skills: pyspark,agile,scala,aws,data,big data