Senior Data Platform Engineer

Relevance Lab; Bengaluru; Hybrid; INR 400,000 - 700,000; Full time

Job summary

Relevance Lab is seeking an experienced Senior Data Platform Engineer to design, build, and operate scalable data platforms in a cloud-first environment. The role emphasizes Python, PySpark, Snowflake, Kafka, and Kubernetes, with a focus on reliable and cost-efficient data infrastructure spanning ingestion, processing, storage, and access.

You will collaborate across data engineering, analytics, security, and application teams to deliver production-grade data capabilities for BI, product

Qualifications

• Experience in data platform engineering and production systems.
• Strong Python and data processing skills.
• Production data pipelines with batch and streaming.
• Ability to learn new technologies independently.

Responsibilities

• Design, develop, and operate scalable data platforms.
• Build ETL/ELT pipelines for batch and streaming data.
• Implement data quality checks and monitoring.
• Collaborate with analytics, security, and application teams.
• Optimize performance and cost of data workloads.
• Automate infrastructure with IaC and CI/CD.

Skills

Data engineering

Python

PySpark

Spark SQL

Snowflake

Apache Kafka

Kubernetes

Terraform

Cloud (AWS)

Data pipelines

Tools

Snowflake

Apache Kafka

Apache Spark

PySpark

PostgreSQL

Airflow

Terraform

Docker

Kubernetes

Job description

About the Role

We are looking for an experienced Senior Data Platform Engineer to design, develop, implement, and operate scalable, reliable, secure, and cost-effective data platforms. The ideal candidate will have strong hands-on experience in data engineering, distributed systems, cloud infrastructure, data pipelines, and production software engineering.

The role involves building and maintaining the underlying infrastructure responsible for data ingestion, processing, storage, transformation, and accessibility across enterprise systems. The engineer will work closely with data engineering, analytics, security, compliance, and application engineering teams to deliver high-quality data platform capabilities that support business intelligence, product analytics, operational efficiency, and regulatory reporting.

The ideal candidate should possess strong programming skills, particularly in Python, along with relevant experience in technologies such as PySpark, Snowflake, AWS, Apache Kafka, Kubernetes, and infrastructure automation. Experience with the entire technology stack is not mandatory; however, candidates must demonstrate deep production experience in at least one relevant data or infrastructure domain and the ability to learn new technologies independently.

Key Responsibilities

1. Data Platform Engineering

• Design, develop, and maintain scalable data platform components supporting enterprise data ingestion, movement, processing, storage, and consumption.
• Build and enhance data pipelines for batch processing, streaming, and near-real-time data integration.
• Develop reusable frameworks and automation components for data ingestion, transformation, validation, reconciliation, logging, and error handling.
• Work with data warehouses, data lakes, relational databases, and distributed processing systems to support business-critical data workloads.
• Evaluate suitable technologies and architectural approaches based on scalability, reliability, maintainability, performance, and cost.
• Improve existing data platform architecture and modernize legacy data processing solutions where required.

2. Data Engineering and Distributed Processing

• Develop and optimize data processing solutions using Python, PySpark, Spark SQL, and other suitable programming technologies.
• Design and maintain ETL/ELT pipelines for structured and semi-structured data.
• Implement batch and streaming data processing using Apache Kafka or similar messaging technologies.
• Build efficient data transformation and ingestion workflows that support large data volumes and downstream applications.
• Develop data quality checks, validation mechanisms, reconciliation processes, and exception-handling frameworks.
• Troubleshoot pipeline failures, processing bottlenecks, data inconsistencies, and performance issues.
• Work with cloud data platforms such as Snowflake and data lake technologies to support scalable data storage and processing.

3. Cloud Infrastructure and Platform Automation

• Design and support cloud-based data infrastructure, particularly on AWS.
• Work with cloud storage, compute, networking, and data services to build reliable and cost-efficient platform solutions.
• Use infrastructure-as-code tools such as Terraform or Terragrunt to automate infrastructure provisioning and configuration.
• Work with containerization and orchestration technologies such as Docker and Kubernetes where applicable.
• Develop reusable infrastructure modules, engineering utilities, and automation frameworks to improve developer productivity.
• Implement CI/CD pipelines to automate code validation, testing, deployment, and release management.
• Improve deployment safety through automated validation, version control, rollback mechanisms, and repeatable delivery practices.

4. Software Engineering and Development

• Write clean, modular, testable, maintainable, and production-ready code using Python or another general-purpose programming language.
• Apply sound software engineering principles, including object-oriented design where appropriate, modular architecture, exception handling, and automated testing.
• Participate in code reviews, technical design discussions, and engineering best-practice initiatives.
• Build reusable libraries, frameworks, and utilities that can be adopted across engineering teams.
• Automate repetitive operational tasks and improve the efficiency of development and support workflows.
• Diagnose complex technical issues and implement sustainable solutions rather than relying on temporary fixes.

5. Performance Optimization and Production Operations

• Monitor data platform performance, availability, reliability, and resource utilization.
• Optimize Spark jobs, SQL queries, data models, partitioning strategies, and cloud resource configurations.
• Identify performance bottlenecks and improve processing efficiency and infrastructure cost.
• Implement observability, logging, monitoring, alerting, and operational readiness practices.
• Participate in incident investigation, root-cause analysis, remediation, and recovery planning.
• Design systems with appropriate failure handling, resilience, backup, recovery, and safe deployment practices.
• Take end-to-end ownership of assigned platform components from problem definition and design through implementation, rollout, and ongoing production support.

6. Data Security, Governance, and Compliance

• Incorporate data security, privacy, access control, and governance requirements into platform architecture and implementation.
• Implement appropriate access management, data classification, masking, and protection mechanisms where required.
• Support data lineage, auditability, data quality, and governance automation.
• Work with security, compliance, and business stakeholders to ensure platform solutions meet applicable organizational and regulatory requirements.
• Contribute to the development of reliable data systems operating in regulated or security-sensitive environments.

7. AI-Assisted Engineering and Productivity

• Actively use AI-assisted engineering tools to accelerate technical research, solution design, coding, debugging, testing, and documentation.
• Leverage tools such as GitHub Copilot, ChatGPT, Claude, or equivalent engineering assistants where appropriate.
• Critically validate AI-generated code, recommendations, and documentation before incorporating them into production solutions.
• Use AI responsibly while protecting confidential information, sensitive data, and proprietary code.
• Identify repeatable opportunities to incorporate AI into engineering workflows and improve development speed, quality, and operational effectiveness.

8. Technical Leadership and Collaboration

• Independently drive complex engineering assignments with minimal day-to-day supervision.
• Break ambiguous business or technical problems into clear milestones and deliver actionable solutions.
• Communicate architectural decisions, implementation approaches, technical risks, and delivery status clearly.
• Collaborate with distributed teams across India and US time zones using effective written and asynchronous communication.
• Participate in technical design reviews, documentation, knowledge sharing, and engineering best-practice initiatives.
• Mentor engineers and help improve the technical capability of the wider team where appropriate.

Required Technical Skills

Core Data Engineering

• Strong hands-on experience in data engineering, data platform development, or data infrastructure.
• Data ingestion, ETL/ELT, data transformation, and pipeline orchestration.
• Batch processing and/or streaming data processing.
• Experience with distributed data processing or messaging technologies.
• Production experience in at least one relevant domain, such as data pipelines, data warehouses, data lakes, distributed systems, operational databases, Kubernetes, or cloud infrastructure.

Programming and Software Development

• Strong proficiency in Python or another general-purpose programming language.
• Experience writing, testing, debugging, reviewing, and maintaining production-quality code.
• SQL and data processing fundamentals.
• Automation, version control, code quality, and CI/CD practices.

Cloud and Data Platforms

• Experience with cloud infrastructure, preferably AWS, or equivalent cloud platforms.
• Experience with one or more relevant data technologies such as Snowflake, Apache Kafka, Apache Spark, PySpark, PostgreSQL, or data lake technologies.
• Understanding of distributed systems, scalability, fault tolerance, and system design.

Production Engineering

• Experience operating production systems and resolving technical incidents.
• Knowledge of monitoring, observability, logging, deployment safety, and recovery mechanisms.
• Understanding of performance optimization, resource utilization, and infrastructure cost management.
• Ability to design reliable, secure, maintainable, and scalable solutions.

AI Tools

• Active and responsible use of AI-assisted development tools in engineering workflows.
• Ability to validate AI-generated output and independently assess its correctness and suitability.

Good-to-Have Skills

• Snowflake data warehousing, data modeling, and performance optimization.
• Apache Kafka and event-driven data architectures.
• AWS data services, cloud-native data platforms, and Lakehouse architectures.
• Databricks, Delta Lake, Spark SQL, and PySpark optimization.
• Apache Airflow or other workflow orchestration technologies.
• Terraform/Terragrunt and infrastructure as code.
• Docker, Kubernetes, Jenkins, Git, and automated CI/CD.
• PostgreSQL and other operational database technologies.
• dbt, Dagster, time-series databases, and open table formats.
• Change Data Capture (CDC), high-volume ingestion, and event streaming.
• Data governance, data classification, masking, lineage, identity integrations, and access control.
• Experience building reusable internal platforms, infrastructure modules, or developer tooling.
• Experience in fintech, banking, payments, financial services, regulatory reporting, or other regulated environments.

Experience and Qualifications

• Senior-level experience in data engineering, data infrastructure, platform engineering, backend engineering, or distributed systems.
• Typically 10+ years of overall experience is a suggested sourcing benchmark, with substantial hands-on experience designing and operating production systems.
• Demonstrated ownership of technical projects from requirements and design through implementation, deployment, and ongoing support.
• Strong problem-solving and system-design capabilities, including the ability to evaluate architectural trade-offs involving scale, reliability, security, cost, and maintainability.
• Proven ability to learn unfamiliar technologies and become productive without extensive day-to-day supervision.
• Excellent written and verbal communication skills.
• Ability to collaborate effectively with geographically distributed engineering teams.