Senior Data Engineer

SourceMash; Sahibzada Ajit Singh Nagar; On-site; INR 1,800,000 - 2,800,000; Full time

Job summary

SourceMash in Mohali is hiring a Senior Data Engineer to design, build, and manage scalable data solutions. You will own end-to-end data workflows, ensure data quality, and collaborate with AI teams to deliver data products.

The role requires strong Python expertise, hands-on cloud experience (AWS/Azure/GCP), and proficiency with modern data processing frameworks and data warehouses.

Qualifications

• 5+ years hands-on experience in Data Engineering with production pipelines.
• Strong expertise in Python for data engineering.
• Experience with cloud platforms such as AWS, Azure, or Google Cloud.
• Strong knowledge of cloud data warehouses including Snowflake, BigQuery, Redshift, or Synapse Analytics.
• Hands-on experience with distributed data processing frameworks such as Spark, Apache Beam, Databricks, or similar technologies.
• Strong understanding of data partitioning, shuffling, windowing operations, parallel processing, and memory optimization.
• Experience with Docker, Git workflows, automated testing, CI/CD pipelines, and deployment processes.
• Experience implementing Infrastructure as Code solutions using Terraform, CloudFormation, ARM, or Bicep.
• Strong understanding of data quality, reliability, and production system design principles.
• Experience with ML/NLP engineering, LLMs, RAG, vector databases, or MLOps is preferred.
• Ability to design scalable architectures while ensuring downstream system stability and performance.
• Maintain records of the conversations with the customer and analyze the data.
• Handling customer queries on Chat and E-mails.

Responsibilities

• Build and manage production data pipelines with a focus on scalability and reliability.
• Develop Python-based data engineering solutions including API integrations, automation, testing, and pipeline orchestration.
• Design and optimize distributed data processing workflows using technologies such as Spark, Apache Beam, or Databricks.
• Implement data partitioning, parallel processing, memory optimization, and performance tuning strategies.
• Work with cloud data platforms including AWS, Azure, or Google Cloud.
• Design and maintain data warehouse solutions using Snowflake, BigQuery, Redshift, or Synapse Analytics.
• Implement Infrastructure as Code using Terraform, CloudFormation, ARM/Bicep, or equivalent tools.
• Develop CI/CD workflows using Git, Docker, Jenkins, GitHub Actions, or Azure DevOps.
• Ensure data correctness through idempotent pipelines, deterministic deduplication, stable key generation, and validation frameworks.
• Support AI/ML initiatives including NLP, LLM workflows, RAG systems, vector search, and MLOps processes (preferred).
• Optimize LLM-based workflows through rate limiting, concurrency management, cost optimization, and latency improvements.

Skills

Python

Spark

Apache Beam

Databricks

Docker

Git workflows

CI/CD pipelines

Terraform

CloudFormation

ARM/Bicep

AWS

Azure

GCP

Snowflake

BigQuery

Redshift

Synapse

SQL

MLOps

LLMs

Vector databases

Data quality

Distributed processing

Tools

Docker

GitHub Actions

Terraform

CloudFormation

ARM/Bicep

Job description

Let's grow together Senior Data Engineer Education

Graduation

Experience

5+ years

Total Openings

1 New Opening

Location

Mohali

Job Description

We are looking for an experienced Senior Data Engineer to design, build, and manage scalable, reliable, and high-performance data solutions. The ideal candidate will have strong expertise in developing production-grade data pipelines, cloud data platforms, distributed processing systems, and modern data engineering practices. This role involves owning end-to-end data workflows, ensuring data quality and reliability, optimizing cloud infrastructure, and collaborating with engineering, analytics, and AI teams to deliver impactful data products.

Key Responsibilities

• Build and manage production data pipelines with a focus on scalability and reliability.
• Develop Python-based data engineering solutions including API integrations, automation, testing, and pipeline orchestration.
• Design and optimize distributed data processing workflows using technologies such as Spark, Apache Beam, or Databricks.
• Implement data partitioning, parallel processing, memory optimization, and performance tuning strategies.
• Work with cloud data platforms including AWS, Azure, or Google Cloud.
• Design and maintain data warehouse solutions using Snowflake, BigQuery, Redshift, or Synapse Analytics.
• Implement Infrastructure as Code using Terraform, CloudFormation, ARM/Bicep, or equivalent tools.
• Develop CI/CD workflows using Git, Docker, Jenkins, GitHub Actions, or Azure DevOps.
• Ensure data correctness through idempotent pipelines, deterministic deduplication, stable key generation, and validation frameworks.
• Support AI/ML initiatives including NLP, LLM workflows, RAG systems, vector search, and MLOps processes (preferred).
• Optimize LLM-based workflows through rate limiting, concurrency management, cost optimization, and latency improvements.

Requirements

• 5+ years of hands-on experience in Data Engineering with production data pipelines.
• Strong expertise in Python for data engineering applications.
• Experience with cloud platforms such as AWS, Azure, or Google Cloud.
• Strong knowledge of cloud data warehouses including Snowflake, BigQuery, Redshift, or Synapse Analytics.
• Hands-on experience with distributed data processing frameworks such as Spark, Apache Beam, Databricks, or similar technologies.
• Strong understanding of data partitioning, shuffling, windowing operations, parallel processing, and memory optimization.
• Experience with Docker, Git workflows, automated testing, CI/CD pipelines, and deployment processes.
• Experience implementing Infrastructure as Code solutions using Terraform, CloudFormation, ARM, or Bicep.
• Strong understanding of data quality, reliability, and production system design principles.
• Experience with ML/NLP engineering, LLMs, RAG, vector databases, or MLOps is preferred.
• Ability to design scalable architectures while ensuring downstream system stability and performance.
• Maintain records of the conversations with the customer and analyze the data.
• Handling customer queries on Chat and E-mails.