Senior Data Engineer

SourceMash; Mohali; On-site; INR 2,500,000 - 5,000,000; Full time

Job summary

SourceMash in Mohali is seeking an experienced Senior Data Engineer to design, build, and manage scalable data solutions. You will own end-to-end data workflows, implement production-grade pipelines, and work across analytics, AI, and engineering teams.

The role requires strong Python, cloud data platforms (AWS/Azure/GCP), and experience with Spark, Databricks, Snowflake or BigQuery. You will implement IaC, CI/CD, and data quality controls while optimizing cost and latency.

Qualifications

• 5+ years of hands-on data engineering experience with production pipelines.
• Proficient in Python for data engineering tasks.
• Experience with AWS, Azure, or Google Cloud.
• Knowledge of cloud data warehouses (Snowflake, BigQuery, Redshift, Synapse).
• Hands-on with Spark, Apache Beam, Databricks or similar.
• Familiar with Docker, Git, CI/CD and deployment processes.
• Experience IaC using Terraform, CloudFormation, ARM, or Bicep.
• Understanding data quality, reliability, and production system design.
• Experience with ML/NLP workflows, LLMs, or MLOps preferred.

Responsibilities

• Build and manage production data pipelines with scalability and reliability.
• Develop Python-based data engineering solutions including API integrations, automation, testing, and pipeline orchestration.
• Design and optimize distributed data processing workflows using Spark, Apache Beam, or Databricks.
• Implement data partitioning, parallel processing, memory optimization, and performance tuning.
• Work with cloud data platforms including AWS, Azure, or Google Cloud.
• Design and maintain data warehouse solutions using Snowflake, BigQuery, Redshift, or Synapse Analytics.
• Implement Infrastructure as Code using Terraform, CloudFormation, ARM/Bicep, or equivalent tools.
• Develop CI/CD workflows using Git, Docker, Jenkins, GitHub Actions, or Azure DevOps.
• Ensure data correctness through idempotent pipelines, deduplication, stable key generation, and validation frameworks.
• Support AI/ML initiatives including NLP, LLM workflows, RAG systems, vector search, and MLOps processes (preferred).
• Optimize LLM-based workflows through rate limiting, concurrency management, cost optimization, and latency improvements.

Skills

Python

Spark

Databricks

Terraform

Docker

CI/CD

Cloud platforms

Snowflake/BigQuery

LLMs

Tools

Docker

Git

Terraform

CI/CD tooling

Job description

Let's grow together Senior Data Engineer Education

Graduation

Experience

5+ years

Total Openings

1 New Opening

Location

Mohali

Job Description

We are looking for an experienced Senior Data Engineer to design, build, and manage scalable, reliable, and high-performance data solutions. The ideal candidate will have strong expertise in developing production-grade data pipelines, cloud data platforms, distributed processing systems, and modern data engineering practices. This role involves owning end-to-end data workflows, ensuring data quality and reliability, optimizing cloud infrastructure, and collaborating with engineering, analytics, and AI teams to deliver impactful data products.

Key Responsibilities

• Build and manage production data pipelines with a focus on scalability and reliability.
• Develop Python-based data engineering solutions including API integrations, automation, testing, and pipeline orchestration.
• Design and optimize distributed data processing workflows using technologies such as Spark, Apache Beam, or Databricks.
• Implement data partitioning, parallel processing, memory optimization, and performance tuning strategies.
• Work with cloud data platforms including AWS, Azure, or Google Cloud.
• Design and maintain data warehouse solutions using Snowflake, BigQuery, Redshift, or Synapse Analytics.
• Implement Infrastructure as Code using Terraform, CloudFormation, ARM/Bicep, or equivalent tools.
• Develop CI/CD workflows using Git, Docker, Jenkins, GitHub Actions, or Azure DevOps.
• Ensure data correctness through idempotent pipelines, deterministic deduplication, stable key generation, and validation frameworks.
• Support AI/ML initiatives including NLP, LLM workflows, RAG systems, vector search, and MLOps processes (preferred).
• Optimize LLM-based workflows through rate limiting, concurrency management, cost optimization, and latency improvements.

Requirements

• 5+ years of hands-on experience in Data Engineering with production data pipelines.
• Strong expertise in Python for data engineering applications.
• Experience with cloud platforms such as AWS, Azure, or Google Cloud.
• Strong knowledge of cloud data warehouses including Snowflake, BigQuery, Redshift, or Synapse Analytics.
• Hands-on experience with distributed data processing frameworks such as Spark, Apache Beam, Databricks, or similar technologies.
• Strong understanding of data partitioning, shuffling, windowing operations, parallel processing, and memory optimization.
• Experience with Docker, Git workflows, automated testing, CI/CD pipelines, and deployment processes.
• Experience implementing Infrastructure as Code solutions using Terraform, CloudFormation, ARM, or Bicep.
• Strong understanding of data quality, reliability, and production system design principles.
• Experience with ML/NLP engineering, LLMs, RAG, vector databases, or MLOps is preferred.
• Ability to design scalable architectures while ensuring downstream system stability and performance.
• Maintain records of the conversations with the customer and analyze the data.
• Handling customer queries on Chat and E-mails.