Data Engineer

Lancesoft; Bengaluru, Chennai District; On-site; INR 1,800,000 - 3,000,000; Full time

Job summary

Lancesoft is seeking a Senior Python Data Engineer to design, build, and lead enterprise-scale AI/ML data pipelines. You will enable machine learning pipelines, AI-driven analytics, and data platforms while mentoring a team of engineers.

You will collaborate with data scientists to productionize models, develop scalable Python-based processing frameworks, and ensure secure, compliant data architectures across cloud or on-prem environments.

Qualifications

• 6–8 years of experience in data engineering with strong Python expertise.
• Proven experience supporting Artificial Intelligence and Machine Learning workloads.
• Strong hands-on experience with: Python (Pandas, NumPy, PySpark, or similar); SQL and data modeling; ETL/ELT pipeline development.
• Experience working with large-scale datasets.
• Solid understanding of ML lifecycle and data requirements for AI models.
• Prior experience leading or mentoring engineers.

Responsibilities

• Design, develop, and maintain scalable data pipelines to support AI and ML use cases.
• Build efficient ETL/ELT workflows for structured and unstructured data.
• Prepare, transform, and optimize datasets for training, testing, and deploying ML models.
• Collaborate closely with data scientists to productionize ML and AI models.
• Develop robust, scalable Python-based data processing frameworks.
• Optimize data processing for performance, reliability, and scalability.
• Design and implement cloud-based or on-prem data architectures for AI/ML.
• Work with big data and distributed processing frameworks.
• Ensure data security, governance, and compliance across pipelines.
• Lead, mentor, and guide a team of data engineers.
• Provide technical direction, task planning, and delivery oversight.
• Participate in architecture reviews, design discussions, and technology evaluations.
• Act as a bridge between engineering, data science, and business teams.

Skills

Python

ETL/ELT

SQL

Data modeling

Big data

Pandas/NumPy

PySpark

Cloud platforms

Airflow

ML lifecycle

Tools

PySpark

Spark

Airflow

Job description

Role & responsibilities

Senior Python Data Engineer AI & Machine Learning

Experience

68 Years

Job Summary

We are seeking an experienced Python Data Engineer who can design, build, and lead AI/ML data engineering solutions at an enterprise scale. The candidate will play a key role in enabling machine learning pipelines, AI-driven analytics, and data platforms, while also leading and mentoring a team of engineers.

Key Responsibilities

Data Engineering & AI/ML Enablement

• Design, develop, and maintain scalable data pipelines to support AI and Machine Learning use cases.
• Build efficient ETL/ELT workflows for structured and unstructured data.
• Prepare, transform, and optimize datasets for training, testing, and deploying ML models.
• Collaborate closely with data scientists to productionize ML and AI models.

Python Development

• Develop robust, scalable, and reusable Pythonbased data processing frameworks.
• Optimize data processing for performance, reliability, and scalability.
• Ensure high code quality through code reviews, testing, and best practices.

Platform & Architecture

• Design and implement cloudbased or onprem data architectures for AI/ML.
• Work with big data and distributed processing frameworks where applicable.
• Ensure data security, governance, and compliance across pipelines.

Leadership & Team Development

• Lead, mentor, and guide a team of data engineers.
• Provide technical direction, task planning, and delivery oversight.
• Participate in architecture reviews, design discussions, and technology evaluations.
• Act as a bridge between engineering, data science, and business teams.

Required Skills & Qualifications

Mandatory Skills

• 6–8 years of experience in data engineering with strong Python expertise.
• Proven experience supporting Artificial Intelligence and Machine Learning workloads.
• Strong handson experience with:
  • Python (Pandas, NumPy, PySpark, or similar)
  • SQL and data modeling
  • ETL/ELT pipeline development
• Experience working with largescale datasets.
• Solid understanding of ML lifecycle and data requirements for AI models.
• Prior experience leading or mentoring engineers.

Technical Skills

• Python for data engineering
• SQL & relational databases
• Big data frameworks (Spark/PySpark – preferred)
• Cloud platforms (AWS/Azure/GCP – preferred)
• Workflow orchestration (Airflow or similar – preferred)
• REST APIs and data integrations
• CI/CD and basic DevOps practices

Nice to Have

• Experience with MLOps practices and tools.
• Exposure to Deep Learning or Generative AI workflows.
• Knowledge of data governance, lineage, and monitoring.
• Agile/Scrum delivery experience.