Lead Data Engineer

Minfy Technologies; Chennai District; On-site; INR 4,000,000 - 7,000,000; Full time

Job summary

Minfy Technologies in Chennai, India seeks a Lead Data Engineer with 8-10 years IT experience and 4-5 years on AWS data services. The role focuses on designing and implementing robust, scalable AWS-based data solutions with Redshift as a core competency.

You will architect data pipelines, lakehouse schemas, and governance while mentoring engineers and collaborating with analysts and stakeholders. The position emphasizes cloud-native, cost-efficient data platforms.

Qualifications

• 8-10 years overall IT experience with 4-5 years on AWS data services.
• Experience designing and developing data applications on AWS Cloud.
• Deep expertise in Amazon Redshift including modelling, performance, Spectrum, and data sharing.

Responsibilities

• Lead design, development, and optimization of large-scale data warehousing on Redshift.
• Architect end-to-end batch and streaming pipelines using Glue, EMR, Spark, MSK/Kafka, and Kinesis.
• Design lakehouse architectures on S3 using Apache Iceberg integrated with Redshift and Athena.
• Orchestrate workflows with Step Functions and MWAA.

Skills

Amazon Redshift

SQL

Data Warehousing

Apache Spark

Kafka/MSK

AWS Glue

EMR

S3/ Lake Formation

Cloud Architecture

Python scripting

Education

Bachelor's degree in Computer Science/IT/Data Analytics

Tools

AWS CloudFormation

Terraform

Apache Airflow (MWAA)

Kinesis

Iceberg

Job description

Job Title: Lead Data Engineer - AWS (Redshift)

Experience: 8-10 years of overall IT experience, with a minimum of 4-5 years focused on AWS data services, including strong hands-on experience with Amazon Redshift.

---------------------------------------------------------------

ABOUT THE ROLE

We are seeking a highly skilled and experienced Lead Data Engineer to join our data and analytics team. The ideal candidate will have deep technical expertise in AWS data services - with Amazon Redshift as a core competency - and will be responsible for designing, developing, and implementing robust, scalable, and insightful data-intensive solutions on the AWS Cloud.

This role requires a strong foundation in data engineering, advanced SQL skills, and extensive experience across the AWS analytics ecosystem, including Amazon Redshift, AWS Glue, Amazon EMR, Apache Spark, Apache Kafka (Amazon MSK), and Apache Iceberg. You will play a crucial role in building and evolving an AWS-native cloud data platform.

KEY RESPONSIBILITIES

• Lead the design, development, and optimisation of large-scale data warehousing solutions on Amazon Redshift, including schema design, distribution/sort key strategy, workload management (WLM), Redshift Spectrum, and query performance tuning.
• Architect and build end-to-end batch and streaming data pipelines using AWS Glue, Amazon EMR, Apache Spark, Amazon MSK/Kafka, and Amazon Kinesis.
• Design and implement lakehouse architectures on Amazon S3 using open table formats such as Apache Iceberg, integrated with Redshift and Athena.
• Orchestrate complex data workflows using AWS Step Functions and Amazon MWAA (Managed Workflows for Apache Airflow).
• Oversee and contribute to the development, deployment, and lifecycle management of data applications on AWS, ensuring reliability, scalability, and cost efficiency.
• Establish data quality, data governance, security, and access-control standards (Lake Formation, IAM, encryption at rest/in transit).
• Mentor and provide technical leadership to a team of data engineers; conduct design and code reviews.
• Collaborate with data analysts, data scientists, and business stakeholders to translate business requirements into technical solutions.
• Implement monitoring, alerting, and observability for data pipelines using Amazon CloudWatch and related tooling.

REQUIRED SKILLS AND QUALIFICATIONS

• Bachelor's degree in Computer Science, Information Technology, Data Analytics, or a related field.
• 8-10 years of overall IT experience, with 4-5 years of hands-on experience designing and developing data applications on AWS Cloud.
• Deep expertise in Amazon Redshift, including data modelling, performance optimisation, RA3 architecture, Redshift Spectrum, materialised views, and data sharing.
• Strong hands-on experience across AWS data and analytics services, including but not limited to:

• Data Warehousing & Query: Amazon Redshift, Amazon Athena
• Data Integration & Processing: AWS Glue (ETL, Data Catalog, Crawlers), Amazon EMR, Apache Spark, AWS Lambda
• Streaming: Amazon Kinesis, Amazon MSK (Apache Kafka)
• Storage & Lakehouse: Amazon S3, Apache Iceberg, AWS Lake Formation
• Orchestration: AWS Step Functions, Amazon MWAA (Apache Airflow)
• Operations & Monitoring: Amazon CloudWatch, AWS CloudTrail

• Advanced SQL skills and strong experience in dimensional modelling and data warehouse design.
• Proven ability to translate business requirements into technical solutions.
• Excellent analytical, problem-solving, and critical‑thinking skills.
• Strong communication and interpersonal skills, with the ability to collaborate effectively with technical and non‑technical stakeholders.
• Experience working in an Agile development methodology.
• Ability to work independently, manage multiple priorities, and meet tight deadlines.

PREFERRED SKILLS (NICE TO HAVE)

• Proficiency in Python or other scripting languages for data manipulation and automation.
• Experience with infrastructure‑as‑code (AWS CloudFormation, Terraform, or AWS CDK) and CI/CD for data pipelines.
• AWS certifications such as AWS Certified Data Engineer - Associate or AWS Certified Solutions Architect.
• Experience with other hyperscalers (Azure, GCP), demonstrating a broad understanding of data engineering.
• Exposure to dbt, data observability tooling, or modern data stack components.