Lead Data Engineer

NAM Info; Pune District; On-site; INR 4,000,000 - 7,000,000; Full time

Job summary

NAM Info in Pune is seeking a Lead Data Engineer to design and develop scalable data pipelines and integrations to support growing data volumes. You will work with analytics and business teams to build pipelines and data models feeding BI and visualization tools.

Responsibilities include building end-to-end ETL/ELT workflows, optimizing Spark and Databricks workloads, ensuring data quality, and collaborating with stakeholders to deliver data solutions aligned with business goals.

Qualifications

• Bachelor's degree in Computer Science, Engineering, or related field.
• 8+ years of experience in data engineering or related roles.
• Proficiency in Python, Java, or Scala.
• Strong SQL skills and experience with relational databases (e.g., MySQL, PostgreSQL).
• Experience with data warehousing concepts and technologies (e.g., Snowflake, Redshift).
• Familiarity with big data processing frameworks (e.g., Apache Spark, Hadoop).
• Hands-on experience with ETL tools and data integration platforms.
• Knowledge of cloud platforms such as AWS, Azure, or Google Cloud Platform.
• Understanding of data modeling principles and data warehousing design patterns.
• Excellent problem-solving skills and attention to detail.
• Strong communication and collaboration skills, with the ability to work effectively in a team environment

Responsibilities

• Data Pipeline Development: Design, develop, and maintain scalable batch and streaming data pipelines using Apache Spark (PySpark/Scala) and Databricks.
• Data Modeling & Analytics Enablement: Design and maintain efficient data models, schemas, and curated datasets for analytics and BI.
• Data Integration: Integrate data from multiple sources including relational databases, APIs, and flat files.
• Performance Optimization: Identify and resolve bottlenecks in Spark jobs and data storage layers.
• Data Quality & Governance: Implement data quality checks and governance standards to ensure trustworthy data.
• Collaboration & Stakeholder Engagement: Work with analysts, scientists, and business teams to deliver data solutions.
• Documentation & Best Practices: Document pipelines and designs; follow CI/CD and deployment best practices.
• Continuous Improvement: Drive automation and improvements to increase reliability and scalability.

Skills

Python

Java

Scala

SQL

Data modeling

Problem solving

Communication

Teamwork

Education

Bachelor's degree in Computer Science, Engineering, or related field

Tools

Apache Spark

Databricks

ETL tools

Snowflake

Redshift

AWS

Azure

GCP

Job description

Job Title: Lead Data Engineer

Experience: 8+ Years

Location : Pune

Employment Type: Full-time

Job Description:

We are looking for a Lead Data Engineer who is responsible for the design and development of scalable data pipelines and integrations to support continual increases in data volume and complexity. Work with analytics and business teams to understand their needs, create pipelines, improve data models that feed BI and visualization tools.

Key Responsibilities:

Data Pipeline Development
Design, develop, and maintain scalable batch and streaming data pipelines using Apache Spark (PySpark/Scala) and Databricks. Build end-to-end ETL/ELT workflows for ingesting, transforming, and validating data from diverse source systems while ensuring data accuracy, reliability, and performance.

Data Modeling & Analytics Enablement
Design and maintain efficient data models, schemas, and curated datasets that support business analytics, reporting, and visualization tools. Optimize data structures for performance, scalability, and cost across lakehouse and data warehouse platforms.

Data Integration
Integrate data from multiple internal and external sources, including relational databases, APIs, flat files, and streaming sources. Ensure seamless and reliable data movement across cloud platforms, data lakes, and analytics systems.

Performance Optimization
Identify and resolve performance bottlenecks in Spark jobs, Databricks workloads, and data storage layers. Tune Spark configurations, optimize queries, and improve pipeline efficiency to support large-scale data processing.

Data Quality & Governance
Implement data quality checks, validation rules, and governance standards to ensure trustworthy data. Monitor data quality metrics and proactively address data issues in collaboration with stakeholders.

Collaboration & Stakeholder Engagement
Work closely with data analysts, data scientists, and business teams to understand requirements and deliver data solutions aligned with business objectives. Partner with platform and cloud teams to ensure architectural consistency and best practices.

Documentation & Best Practices
Document data pipelines, data models, and technical designs. Follow best practices for software development, version control, CI/CD, and deployment in distributed data environments.

Continuous Improvement
Stay current with emerging data engineering technologies, Spark and Databricks enhancements, and cloud data platform innovations. Drive automation and process improvements to increase reliability, scalability, and developer productivity.

Role & responsibilities

Required Skills and Qualifications:

• Bachelor's degree in Computer Science, Engineering, or related field.
• 8+ years of experience in data engineering or related roles.
• Proficiency in programming languages such as Python, Java, or Scala.
• Strong SQL skills and experience with relational databases (e.g., MySQL, PostgreSQL).
• Experience with data warehousing concepts and technologies (e.g., Snowflake, Redshift).
• Familiarity with big data processing frameworks (e.g., Apache Spark, Hadoop).
• Hands-on experience with ETL tools and data integration platforms.
• Knowledge of cloud platforms such as AWS, Azure, or Google Cloud Platform.
• Understanding of data modeling principles and data warehousing design patterns.
• Excellent problem-solving skills and attention to detail.
• Strong communication and collaboration skills, with the ability to work effectively in a team environment