Data Engineer -AI

SP Software; India; On-site; INR 1,200,000 - 1,800,000; Full time

Job summary

SP Software is seeking a Data Engineer with Gen AI expertise to architect and operate data pipelines across multiple Indian locations including Bangalore, Chennai, Hyderabad, and more.

You will build real-time streaming data solutions, work with PySpark, Snowflake, Kafka, and agentic AI frameworks, and deploy serverless components on Azure. The role expects 5.5+ years of experience, with 6+ years preferred, in a telecom data environment.

Qualifications

• 5.5+ years of data engineering experience, 6+ years preferred.
• Hands-on with PySpark, Spark SQL, and Structured Streaming.
• Experience with Snowflake and Snowpark/Snowpipe Streaming.
• Kafka & Real-time streaming pipelines development.
• Familiarity with agentic AI frameworks (Semantic Kernel, LangChain, AutoGen, CrewAI).
• Azure Functions / serverless deployments, and DevOps practices (Git, CI/CD).
• Telecom domain data/Telemetry experience is a plus.

Responsibilities

• Design and implement data pipelines for large-scale telemetry data.
• Develop real-time streaming solutions using Kafka and Spark Structured Streaming.
• Build data models in Snowflake and orchestrate with Airflow/dbt.
• Develop Gen AI-enabled data processing components and workflows.
• Deploy serverless functions on Azure and manage CI/CD pipelines.
• Create KPI dashboards and automate reporting processes.

Skills

Data engineering

Gen AI

Real-time data processing

Tools

PySpark

Spark SQL

Structured Streaming

Python

Snowflake

Snowpark

Snowpipe Streaming

Kafka

LangChain

Semantic Kernel

AutoGen

CrewAI

Azure Functions

Serverless

Cortex Analyst

Cortex Agents

ADF

Airflow

dbt

Git

CI/CD

Azure DevOps

GitHub Actions

Job description

Job Position: Data Engineer + Gen AI
Location: Bangalore, Chennai, Hyderabad, Noida, Gurgaon, Kochi, Trivandrum, Pune
Experience: 5.5+ Years (6+ Years Preferred)

Key Responsibilities

• Data bricks (PySpark, Spark SQL, Structured Streaming)
• Python
• Snowflake (SQL, Snowpark, Snowpipe Streaming, Data Aggregations, ELT)
• Kafka & Real-time Streaming Pipelines
• Agentic AI Frameworks (Semantic Kernel / LangChain / AutoGen / CrewAI)
• Azure Functions / Serverless Deployments
• KPI Dashboard Automation (Cortex Analyst/Cortex Agents preferred)
• ADF / Airflow / dbt
• DevOps (Git, CI/CD, Azure DevOps/GitHub Actions)
• Telecom Domain experience with Network KPI/Telemetry data preferred