Senior Software Engineer

SourcingXPress; Hyderabad; On-site; INR 4,500,000 - 5,200,000; Full time

Job summary

SourcingXPress in Hyderabad is seeking a seasoned Senior Software Engineer to join the Data Platform team. You will be working on enterprise-scale data platforms using technologies like Apache Airflow and Apache Spark, focusing on running data pipelines and supporting internal AI agents. The ideal candidate has over 5 years in software engineering, deep expertise in distributed systems, and strong AWS skills. You will be responsible for enhancing the data platform's reliability, performance, and scalability while optimizing costs.

Qualifications

• 5+ years of software engineering experience with a focus on data infrastructure, platform engineering, and distributed systems.
• Strong proficiency in at least one programming language - Python/Go/Scala.
• Deep understanding of Apache Spark architecture and experience running Spark workloads in production.

Responsibilities

• Partner with data engineering teams to translate requirements into data platform capabilities.
• Build tooling and automation to improve developer experience.
• Establish monitoring and alerting practices for platform performance.

Skills

Software engineering experience

Python

Apache Spark

AWS services

Data engineering concepts

Kubernetes

Tools

Apache Airflow

EMR

EKS

DataHub

Job description

Job Description

We're seeking an experienced Senior Software Engineer to join our Data Platform team. You will work on enterprise‑scale data platforms including Apache Airflow for workflow orchestration, Apache Spark on EMR on EKS for distributed data processing and DataHub for data certification. You would be working on the internal data platform to efficiently run the company‑wide data pipelines, analytics jobs and power internal AI agents with trusted data products.

Salary Range

₹45-52 Lacs PA

Required Experience

• 5+ years of software engineering experience with a focus on data infrastructure, platform engineering, and distributed systems
• Strong proficiency in at least one programming language - Python/Go/Scala
• Deep understanding of Apache Spark architecture and experience running Spark workloads in production
• Solid experience with AWS services, particularly EMR, EKS, S3, etc.
• Experience with Data Platform infrastructure on Kubernetes and container orchestration
• Strong understanding of data engineering concepts, workflows, and best practices
• Proven ability to debug complex issues in distributed systems
• Knowledge of data quality frameworks and tooling (Great Expectations, Deequ, Monte Carlo, etc.)
• Experience with data lineage tools and metadata management (Apache Atlas, DataHub, OpenLineage, etc.)
• Understanding of data governance principles and compliance requirements
• Experience with Apache Airflow setup
• AWS Certification is a plus (but not required)

Responsibilities

• Partner with data engineering teams to understand requirements and translate them into data platform capabilities
• Build tooling and automation to improve developer experience and reduce operational overhead
• Establish monitoring, alerting, and observability practices for platform health and performance
• Drive platform upgrades, migrations, and adoption of new technologies
• Own and maintain the Apache Airflow orchestration platform, ensuring reliable DAG execution for data teams across the organization
• Manage and optimize the Apache Spark platform on EMR on EKS, supporting large-scale data processing workloads
• Design and implement platform improvements to enhance reliability, scalability, and performance
• Troubleshoot complex distributed systems issues and provide timely resolution for platform incidents
• Optimize resource utilization and cost efficiency across AWS infrastructure
• Create and maintain technical documentation, runbooks, and best practices
• Participate in on-call rotation and incident response to ensure platform availability
• Conduct capacity planning and scaling to accommodate organizational growth