Lead Data Engineer

Inferyx Inc; Pune District; On-site; INR 2,500,000 - 4,500,000; Full time

Job summary

Inferyx Inc. seeks a Lead Data Engineer to design and develop scalable data pipelines and integrations for growing data volume and complexity.

You will collaborate with analytics and business teams to understand needs, build pipelines, and improve data models feeding BI and visualization tools. You will lead initiatives across Spark/Databricks, ETL/ELT workflows, data quality and governance, and collaboration with cloud platforms to ensure reliable data movement across lakehouse and data

Qualifications

• Bachelor's degree in CS/Engineering or related field.
• 8+ years data engineering experience.
• Proficiency in Python/Java/Scala.
• Strong SQL and RDBMS experience.
• Experience with cloud platforms (AWS/Azure/GCP).

Responsibilities

• Design, develop, and maintain scalable batch and streaming pipelines using Spark and Databricks.
• Build end-to-end ETL/ELT workflows from diverse sources.
• Design and maintain data models and curated datasets for analytics.
• Optimize data structures for performance and cost across lakehouse and data warehouses.
• Integrate data from multiple sources ensuring reliable data movement across clouds.
• Implement data quality checks and governance standards.
• Collaborate with analysts, scientists, and business teams; document pipelines and designs.

Skills

Python

Java

Scala

SQL

Communication

Collaboration

Problem solving

Education

Bachelors in CS/Engineering

Tools

Apache Spark

Databricks

Snowflake

Redshift

Hadoop

ETL tools

Job description

We are looking for a Lead Data Engineer who is responsible for the design and development of scalable data pipelines and integrations to support continual increases in data volume and complexity. Work with analytics and business teams to understand their needs, create pipelines, and improve data models that feed BI and visualisation tools.

The candidate will have responsibilities across the following functions:

Data Pipeline Development:

• Design, develop, and maintain scalable batch and streaming data pipelines using Apache Spark (PySpark/Scala) and Databricks.
• Build end-to-end ETL/ELT workflows for ingesting, transforming, and validating data from diverse source systems while ensuring data accuracy, reliability, and performance.

Data Modelling and Analytics Enablement:

• Design and maintain efficient data models, schemas, and curated datasets that support business analytics, reporting, and visualisation tools.
• Optimise data structures for performance, scalability, and cost across lakehouse and data warehouse platforms.

Data Integration:

• Integrate data from multiple internal and external sources, including relational databases, APIs, flat files, and streaming sources.
• Ensure seamless and reliable data movement across cloud platforms, data lakes, and analytics systems.

Performance Optimisation:

• Identify and resolve performance bottlenecks in Spark jobs, Databricks workloads, and data storage layers.
• Tune Spark configurations, optimise queries, and improve pipeline efficiency to support large-scale data processing.

Data Quality and Governance:

• Implement data quality checks, validation rules, and governance standards to ensure trustworthy data.
• Monitor data quality metrics and proactively address data issues in collaboration with stakeholders.

Collaboration and Stakeholder Engagement:

• Work closely with data analysts, data scientists, and business teams to understand requirements and deliver data solutions aligned with business objectives.
• Partner with platform and cloud teams to ensure architectural consistency and best practices.

Documentation and Best Practices:

• Document data pipelines, data models, and technical designs.
• Follow best practices for software development, version control, CI/CD, and deployment in distributed data environments.

Continuous Improvement:

• Stay current with emerging data engineering technologies, Spark and Databricks enhancements, and cloud data platform innovations.
• Drive automation and process improvements to increase reliability, scalability, and developer productivity.

Requirements:

• Bachelor's degree in Computer Science, Engineering, or related field.
• 8+ years of experience in data engineering or related roles.
• Proficiency in programming languages such as Python, Java, or Scala.
• Strong SQL skills and experience with relational databases (e. g., MySQL, PostgreSQL).
• Experience with data warehousing concepts and technologies (e. g., Snowflake, Redshift).
• Familiarity with big data processing frameworks (e. g., Apache Spark, Hadoop).
• Hands‑on experience with ETL tools and data integration platforms.
• Knowledge of cloud platforms such as AWS, Azure, or Google Cloud Platform.
• Understanding of data modelling principles and data warehousing design patterns.
• Excellent problem‑solving skills and attention to detail.
• Strong communication and collaboration.