Data Solution Engineer

Robosoft Technologies Inc; Pune District, Mumbai, Bengaluru; Hybrid; INR 2,500,000 - 4,200,000; Full time

Job summary

Robosoft Technologies Inc in Pune, Maharashtra, is seeking a senior data engineer to design and manage data processing pipelines, API development, and scalable Spark-based workloads.

The role requires 5+ years of hands-on experience with PySpark, Scala, Databricks, and cloud data platforms, plus expertise in CI/CD, Airflow, and Terraform to build reliable, cost-efficient data solutions.

Qualifications

• 5+ years in data processing pipelines and API development.
• Strong Python (PySpark) and Scala skills.
• Experience with Spark, Databricks and cloud Spark on GCP Dataproc.
• Familiar with Delta Lake, Iceberg and Hudi table formats.
• Data modeling across SQL and NoSQL databases (Cosmos DB, Redis).
• Building batch and streaming ETL/ELT with Kafka, Pub/Sub and Event Hubs.
• Orchestrate with Airflow on Kubernetes in containerized architectures.
• Infrastructure as Code with Terraform or equivalent.
• Data quality, lineage, monitoring and testing pipelines.
• Observability: logging, metrics, SLAs/SLOs, incident handling.
• ML engineering: feature store, training pipelines, MLflow/MLOps.
• AI-assisted development with GitHub Copilot or Claude Code.
• Nice to have: Adobe Analytics or Google Analytics.

Skills

Python

PySpark

Scala

Apache Spark

Kafka

Databricks

Kubernetes

ETL/ELT design

Tools

Airflow

Terraform

Google Cloud Dataproc

BigQuery

Delta Lake

Iceberg

Hudi

Blob Storage

Job description

• Core experience: Minimum 5 years of hands-on experience in data processing pipeline and API development, with strong proficiency in Python (PySpark) and Scala a must
• Big data & distributed systems: Apache Spark, Databricks, Blob Storage, BigQuery and managed Spark (Dataproc) on GCP; processing structured, semi-structured and multi-modal data (text, image, audio, video)
• Lakehouse & table formats: Delta Lake, Apache Iceberg and Apache Hudi
• Data modelling & stores: Designing data schemas, tables and partitioning strategies across SQL and NoSQL databases such as Cosmos DB and Redis
• Pipelines & orchestration: Building batch and streaming ETL/ELT processes over pub/sub platforms (Kafka, Azure Event Hubs, Google Cloud Pub/Sub), orchestrated with Airflow on Kubernetes, in containerized, scalable and serverless architectures
• Infrastructure as Code: Terraform or equivalent
• Performance & cost optimization: Spark tuning, shuffle optimization, caching, cluster sizing, query optimization, debugging long-running jobs and cloud cost control
• Data quality & lineage: Defining data quality rules and data lineage definitions, and creating monitoring alerts out of them
• Testing for data pipelines: Data validation, regression tests, reconciliation and end-to-end pipeline testing
• Observability & reliability: Centralized logging, metrics, tracing, job monitoring, SLAs/SLOs and incident troubleshooting
• ML engineering: Feature store, feature and training pipelines, model deployment and serving, model monitoring with drift detection and retraining (MLflow / MLOps)
• AI-assisted development: Hands-on experience using tools such as GitHub Copilot, Claude Code or equivalent for code generation, debugging, testing and developer productivity
• Nice to have: Past exposure to web analytics solutions (Adobe Analytics & Google Analytics)