Senior Data Engineer for AI/ML Pipelines (Azure Databricks)

KPIT Technologies Ltd.; Columbus (OH); On-site; USD 110,000 - 160,000; Full time

Benefits offered by this job

Geo Blue PPO and HSA plan.

Dental and Vision plan

401k with employer match

Life and disability insurance

Job summary

KPIT Technologies Ltd. is seeking a data engineer with strong Azure Databricks and Scala/Python expertise to design, build, and deploy scalable data pipelines and ML solutions.

You will own end-to-end delivery from data ingestion to production deployment, with emphasis on data quality, governance, and collaboration with cross-functional teams. The role requires hands-on experience in Spark, Delta Lake, and MLflow, plus a proactive approach to optimizing pipelines and models for real business use

Qualifications

• System Requirements Engineering and traceability across the lifecycle.
• Experience with Azure Databricks and Scala/Python.
• Hands-on data pipelines using Spark and related big-data tech.
• Strong problem solving and communication skills.

Responsibilities

• Designs and automates deployment of our distributed data ingestion pipelines.
• Own end-to-end delivery of AI and ML solutions from problem definition to prod deployment.
• Build and maintain data pipelines using PySpark and Spark in Azure Databricks.
• Designs and implements framework to monitor data quality and integrity issues.
• Implements data governance and metadata management for internal/external users.
• Guide and support team on data engineering, ML, and AI best practices.
• Write production-ready, well-documented code.

Skills

System Requirements Engineering

Requirements Management

Requirements Traceability

Requirements Analysis

Acceptance Criteria Definition

Change Impact Analysis

Stakeholder Management

Communication Skills

Customer Focus

ETL

Data Modeling

Scala

Python

AI Agents

LLMs

Agentic AI Systems

Education

Degree in relevant technical discipline

Tools

Azure Databricks

MLflow

Spark

Delta Lake

Git

CI/CD pipelines

Job description

KPIT Technologies Ltd. is seeking a data engineer with strong Azure Databricks and Scala/Python expertise to design, build, and deploy scalable data pipelines and ML solutions.

You will own end-to-end delivery from data ingestion to production deployment, with emphasis on data quality, governance, and collaboration with cross-functional teams. The role requires hands-on experience in Spark, Delta Lake, and MLflow, plus a proactive approach to optimizing pipelines and models for real business use