Lead Data Engineer (AWS)

Neoware; Chennai District; On-site; INR 2,800,000 - 6,000,000; Full time

Job summary

Neoware is seeking a senior data engineer in Chennai to lead the design and delivery of scalable AWS-based data solutions. You will own data pipelines, governance, and data quality while mentoring a team of engineers.

The role requires deep AWS expertise, strong SQL/NoSQL skills, and hands-on PySpark experience to build data lakes and warehouses that empower analytics and ML initiatives.

Qualifications

• 8+ years of hands-on experience delivering data engineering solutions.
• Experience leading a team of 5+ members, handling stakeholder management, design reviews, code reviews and mentoring.
• Strong hands-on AWS experience with data engineering services (Glue, Lambda, Redshift, EMR, Kinesis).
• Expertise in database concepts, SQL and relational modelling.
• Solid NoSQL understanding and hands-on experience.
• Proficient in building data pipelines with Python and PySpark.
• Strong understanding of data lake/warehouse concepts and modern data architectures.
• Hands-on data modelling (ER and dimensional).
• Experience with data quality management, metadata management, and data governance.
• Familiarity with Git and CI/CD for data components.
• Strong analytical, problem-solving, and collaboration skills.

Responsibilities

• Lead technical architecture and design for data engineering components delivering business solutions.
• Design and implement scalable, secure data solutions on AWS with solid architecture patterns.
• Evaluate and select technology stacks for data engineering solutions.
• Lead, mentor, and manage a team of data engineers to deliver quality on time.
• Oversee data mapping specifications and detailed technical designs.
• Ensure adherence to coding standards and best practices, driving delivery excellence.
• Optimize data solutions for performance, cost, and scalability.
• Ensure data quality and integrity for downstream systems and analytics.
• Plan and manage data projects, balancing scope, risks, and stakeholder expectations.
• Maintain comprehensive technical docs including architectures and data models.
• Collaborate with BI and data science teams to enable advanced analytics.

Skills

Data engineering

AWS

Python

PySpark

SQL

NoSQL

Data modelling

Data pipelines

CI/CD

Team leadership

Stakeholder management

Data quality

Data governance

ETL design

Git

Tools

AWS Glue

AWS Lambda

Amazon Redshift

Amazon EMR

Amazon Kinesis

Job description

• 8+ years of hands-on experience in delivering data engineering solutions
• Experience leading a team of 5+ team members, responsible for technical project management including stakeholder management, design review, code reviews and mentoring
• Strong hands-on experience in AWS, with experience in data engineering related services and technologies including AWS Glue, Lambda, Amazon Redshift, EMR, Amazon Kinesis etc.
• Expertise in database concepts, SQL and relational modelling
• Good understanding and experience working with NoSQL databases
• Strong hands-on experience in implementing data pipelines using Python and PySpark
• Good understanding of data lake and data warehouse concepts, modern data architecture patterns and industry trends in data engineering space
• Hands-on experience in data modelling, with a good understanding of both ER modelling and dimensional modelling
• Good understanding and experience implementing data quality management solutions
• Good understanding of metadata management for data engineering solutions and data platforms
• Good understanding of data governance concepts – including data stewardship, data security and data privacy management
• Experience with version control tools like Git for managing version control for source code
• Experience with DevOps processes, including experience implementing CI/CD pipelines for data engineering solution components
• Strong analytical and problem-solving skills
• Excellent communication and collaboration skills

Responsibilities

• Lead technical architecture and design efforts for data engineering solution components for delivering business solutions
• Design and implement scalable, secure and resilient data engineering solutions on AWS, with a good understanding of data architecture patterns
• Evaluate and select appropriate technology stack for data engineering solution components – cloud native services, open-source tools, data storage, data processing frameworks etc
• Lead, mentor and manage a team of dynamic data engineers for development, testing and implementation of data pipelines ensuing quality, performance and on-time delivery
• Responsible for detailed technical design and data mapping specification
• Ensure adherence to industry best practices, coding standards and guidelines driving continuous improvement in code quality and delivery excellence
• Optimize data solutions for performance, cost-efficiency and scalability
• Ensure data quality and integrity for downstream systems and applications
• Plan, prioritize and manage data engineering projects, ensuring timely delivery with quality
• Identify and mitigate risks, troubleshoot issues, handle customer escalations
• Ensure alignment with business requirements and priorities, working closely with stakeholders – both internal and customer
• Maintain comprehensive technical documentation, including architecture diagrams, high level / low level data integration design, data mapping specifications, database design etc
• Working closely with peers, BI engineers and data scientists to enable advanced analytics, machine learning models and reporting solutions