Senior Data Engineer

Vidpro Consultancy Services; Pune District, Chennai District, Bengaluru; On-site; INR 1,800,000 - 2,800,000; Part time

Job summary

Vidpro Consultancy Services in Chennai seeks a Lead/Senior Data Engineer to design, develop, and deliver scalable data solutions while leading a team of engineers. The role involves hands‑on coding, architectural decisions, and mentoring across cross‑functional teams.

The candidate will architect data pipelines, implement AI/ML‑ready data structures, ensure data quality, and align with governance and security policies. This is a 6‑month contract with aggressive delivery timelines.

Qualifications

• Minimum 6+ years hands-on data engineering experience designing and delivering scalable data platforms.
• Experience with Python, Scala and Spark for large‑scale data processing.
• Proficiency with batch and streaming data processing and data lifecycle management.

Responsibilities

• Lead end‑to‑end design, development, and delivery of data pipelines and software systems.
• Define technical strategy, architecture, and best practices for data engineering.
• Build optimized data pipelines in a cloud environment and manage ETL processes.

Skills

Python

Scala

Spark

SQL

Data pipelines

Agile

Data governance

Distributed systems

Education

Bachelor's degree in CS/Engineering/IS

Tools

Data Lake

Delta Lake

Hive

MySQL

Snowflake

Job description

Job Description:

• Job title : Lead / Senior Data Engineer
• Experience: 6+ years
• Location: Chennai
• Role Type : Contract (6 Months)

Roles and Responsibilities:

Role Overview

• The Lead Data Engineer will be responsible for designing, developing, and delivering high-quality software and data solutions while leading a team of engineers. The role involves hands‑on technical work, architectural decision‑making, mentoring junior developers, and collaborating with cross‑functional teams to ensure successful delivery of scalable data platforms and analytical solutions.

Key Responsibilities

• Lead the end-to-end design, development, and delivery of software systems, data pipelines, and components.
• Define technical strategy, architecture, and best practices for development and data engineering.
• Design and build optimized data pipelines using cutting‑edge technologies in a cloud environment.
• Construct infrastructure for efficient ETL processes from various sources and storage systems.
• Architect, design, and maintain database pipeline architectures, ensuring readiness for AI/ML transformations.
• Lead the implementation of algorithms and prototypes to transform raw data into useful information.
• Review code for quality, scalability, and performance.
• Collaborate with Product Managers, Business Managers, Designers, and QA teams to translate business requirements into technical solutions.
• Develop analytical tools, programs, and reporting mechanisms.
• Create data validation methods and data analysis tools.
• Interpret data trends and patterns to establish operational alerts.
• Conduct complex data analysis and present results effectively.
• Prepare data for prescriptive and predictive modeling.
• Ensure compliance with data governance and security policies.
• Troubleshoot, debug, and resolve complex technical issues.
• Drive continuous improvement in software and data development processes, tools, and methodologies.
• Mentor and guide engineers through code reviews, technical discussions, and training.
• Ensure timely delivery of projects while maintaining high engineering standards.
• Continuously explore opportunities to enhance data quality and reliability.
• Apply strong programming and problem‑solving skills to develop scalable solutions.
• Demonstrate passion for testing strategy, problem‑solving, and continuous learning.
• Willingness to acquire new skills and knowledge.
• Possess a product/engineering mindset to drive impactful data solutions.
• Experience working in distributed environments with global teams.
• Stay current with emerging technologies and industry trends to propose innovative solutions.

Technical Skills and Experience Requirements

• Minimum 6+ years of hands‑on experience designing, building, deploying, testing, maintaining, monitoring, and owning scalable, resilient, and distributed data pipelines.
• High proficiency in Python, Scala, and Spark for applied large‑scale data processing.
• Expertise with big data technologies, including Spark, Data Lake, Delta Lake, and Hive.
• Solid understanding of batch and streaming data processing techniques.
• Proficient knowledge of the Data Lifecycle Management process, including data collection, access, use, storage, transfer, and deletion.
• Expert‑level ability to write complex, optimized SQL queries across extensive data volumes.
• Experience with RDBMS and OLAP databases such as MySQL and Snowflake.
• Familiarity with Agile methodologies.
• Obsession for service observability, instrumentation, monitoring, and alerting.
• Knowledge or experience in architectural best practices for building data lakes.
• Qualifications - Bachelors degree in computer science, Engineering, Information Systems, or related field