Lead Software Engineer

Impetus; Bengaluru; On-site; INR 1,000,000 - 2,000,000; Full time

Job summary

Impetus is seeking a Data Engineer in Bengaluru, India. The ideal candidate will have 6–11 years of experience in the Data Engineering and Big Data ecosystem, with strong expertise in Databricks, Apache Spark, and PySpark. You will design scalable data pipelines, build ETL workflows, and collaborate with data scientists on ML models. Proficiency in SQL and Python, as well as experience with cloud platforms like AWS, Azure, or GCP, are required. This role demands strong problem-solving skills and the ability to thrive in agile environments.

Qualifications

• 6–11 years of experience in Data Engineering / Big Data ecosystem.
• Strong hands-on experience with Databricks, Apache Spark, and PySpark.
• Experience working on cloud platforms (AWS/Azure/GCP).

Responsibilities

• Design and develop scalable data pipelines using Databricks.
• Build and optimize ETL/ELT workflows for large-scale data processing.
• Monitor and optimize performance of data workflows.

Skills

Databricks

Apache Spark

PySpark

SQL

Python

Delta Lake

Machine Learning workflows

Workflow orchestration tools

Cloud platforms (AWS/Azure/GCP)

ML libraries

Job description

• Design and develop scalable data pipelines using Databricks (Spark, PySpark, SQL)
• Build and optimize ETL/ELT workflows for large-scale data processing
• Work with Delta Lake for data management, performance tuning, and reliability
• Collaborate with data scientists to productionize ML models on Databricks
• Implement ML pipelines including data preparation, feature engineering, and model deployment
• Ensure data quality, governance, and security best practices
• Integrate Databricks solutions with cloud platforms (AWS/Azure/GCP)
• Monitor, troubleshoot, and optimize performance of data workflows
• Contribute to CI/CD pipelines and automation for data and ML workflows

Required Skills & Qualifications

• 6–11 years of experience in Data Engineering / Big Data ecosystem
• Strong hands-on experience with Databricks, Apache Spark, and PySpark
• Proficiency in SQL and Python
• Experience with Delta Lake, Unity Catalog (preferred)
• Strong understanding of data modeling and data warehousing concepts
• Experience working on cloud platforms (AWS / Azure / GCP)
• Exposure to Machine Learning workflows (training, deployment, monitoring)
• Familiarity with ML libraries like scikit-learn, MLflow, TensorFlow, or similar
• Experience with workflow orchestration tools (Airflow, ADF, etc.)
• Strong problem-solving and debugging skills

  Good to Have

  • Experience with MLflow for experiment tracking and model management
  • Knowledge of MLOps practices
  • Experience in real-time/streaming pipelines (Kafka, Structured Streaming)
  • Exposure to Data Governance & Security frameworks
  • Certification in Databricks or cloud platforms

  Soft Skills

  • Strong communication and stakeholder management skills
  • Ability to work in agile, fast-paced environments
  • Collaborative mindset with a focus on continuous learning