Sr. Data Platform Engineer(AWS)- Bangalore- Hybrid- Immediate Joiner

Green Hindustan; Bengaluru; Hybrid; INR 1,800,000 - 2,400,000; Full time

Job summary

Green Hindustan in Bengaluru is seeking experienced data platform and data engineering specialists to design, build, and operate scalable data infrastructure across ingestion, processing, and storage pipelines.

You will work with Snowflake, Apache Spark, PySpark, Kafka, and cloud services on AWS, driving reliability, performance, and cost efficiency through automation and CI/CD practices.

Qualifications

• 5+ years in data engineering or platform roles.
• Strong SQL and Python programming skills.
• Experience designing scalable data architectures.
• Familiarity with cloud data services (AWS).

Responsibilities

• Design scalable data ingestion and processing pipelines.
• Develop data quality checks and monitoring.
• Collaborate with cross-functional teams across regions.
• Lead modernization of legacy data processing solutions.

Skills

Data platform

ETL/ELT pipelines

Python

SQL

Spark/PySpark

Kafka

AWS

CI/CD

Distributed systems

Tools

Snowflake

PostgreSQL

Kafka

Docker

Kubernetes

Terraform

Job description

1. Data Platform Engineering

• Design, develop, and maintain scalable data platform components supporting enterprise data ingestion, movement, processing, storage, and consumption.
• Build and enhance data pipelines for batch processing, streaming, and near-real-time data integration.
• Develop reusable frameworks and automation components for data ingestion, transformation, validation, reconciliation, logging, and error handling.
• Work with data warehouses, data lakes, relational databases, and distributed processing systems to support business-critical data workloads.
• Evaluate suitable technologies and architectural approaches based on scalability, reliability, maintainability, performance, and cost.
• Improve existing data platform architecture and modernize legacy data processing solutions where required.

2. Data Engineering and Distributed Processing

• Develop and optimize data processing solutions using Python, PySpark, Spark SQL, and other suitable programming technologies.
• Design and maintain ETL/ELT pipelines for structured and semi-structured data.
• Implement batch and streaming data processing using Apache Kafka or similar messaging technologies.
• Build efficient data transformation and ingestion workflows that support large data volumes and downstream applications.
• Develop data quality checks, validation mechanisms, reconciliation processes, and exception-handling frameworks.
• Troubleshoot pipeline failures, processing bottlenecks, data inconsistencies, and performance issues.
• Work with cloud data platforms such as Snowflake and data lake technologies to support scalable data storage and processing.

3. Cloud Infrastructure and Platform Automation

• Design and support cloud-based data infrastructure, particularly on AWS.
• Work with cloud storage, compute, networking, and data services to build reliable and cost-efficient platform solutions.
• Use infrastructure-as-code tools such as Terraform or Terragrunt to automate infrastructure provisioning and configuration.
• Work with containerization and orchestration technologies such as Docker and Kubernetes where applicable.
• Develop reusable infrastructure modules, engineering utilities, and automation frameworks to improve developer productivity.
• Implement CI/CD pipelines to automate code validation, testing, deployment, and release management.
• Improve deployment safety through automated validation, version control, rollback mechanisms, and repeatable delivery practices.

4. Software Engineering and Development

• Write clean, modular, testable, maintainable, and production-ready code using Python or another general-purpose programming language.
• Apply sound software engineering principles, including object-oriented design where appropriate, modular architecture, exception handling, and automated testing.
• Participate in code reviews, technical design discussions, and engineering best-practice initiatives.
• Build reusable libraries, frameworks, and utilities that can be adopted across engineering teams.
• Automate repetitive operational tasks and improve the efficiency of development and support workflows.
• Diagnose complex technical issues and implement sustainable solutions rather than relying on temporary fixes.

5. Performance Optimization and Production Operations

• Monitor data platform performance, availability, reliability, and resource utilization.
• Optimize Spark jobs, SQL queries, data models, partitioning strategies, and cloud resource configurations.
• Identify performance bottlenecks and improve processing efficiency and infrastructure cost.
• Implement observability, logging, monitoring, alerting, and operational readiness practices.
• Participate in incident investigation, root-cause analysis, remediation, and recovery planning.
• Design systems with appropriate failure handling, resilience, backup, recovery, and safe deployment practices.
• Take end-to-end ownership of assigned platform components from problem definition and design through implementation, rollout, and ongoing production support.

6. Data Security, Governance, and Compliance

• Incorporate data security, privacy, access control, and governance requirements into platform architecture and implementation.
• Implement appropriate access management, data classification, masking, and protection mechanisms where required.
• Support data lineage, auditability, data quality, and governance automation.
• Work with security, compliance, and business stakeholders to ensure platform solutions meet applicable organizational and regulatory requirements.
• Contribute to the development of reliable data systems operating in regulated or security-sensitive environments.

7. AI-Assisted Engineering and Productivity

• Actively use AI-assisted engineering tools to accelerate technical research, solution design, coding, debugging, testing, and documentation.
• Leverage tools such as GitHub Copilot, ChatGPT, Claude, or equivalent engineering assistants where appropriate.
• Critically validate AI-generated code, recommendations, and documentation before incorporating them into production solutions.
• Use AI responsibly while protecting confidential information, sensitive data, and proprietary code.
• Identify repeatable opportunities to incorporate AI into engineering workflows and improve development speed, quality, and operational effectiveness.

8. Technical Leadership and Collaboration

• Independently drive complex engineering assignments with minimal day-to-day supervision.
• Break ambiguous business or technical problems into clear milestones and deliver actionable solutions.
• Communicate architectural decisions, implementation approaches, technical risks, and delivery status clearly.
• Collaborate with distributed teams across India and US time zones using effective written and asynchronous communication.
• Participate in technical design reviews, documentation, knowledge sharing, and engineering best-practice initiatives.
• Mentor engineers and help improve the technical capability of the wider team where appropriate.

Required Technical Skills

Core Data Engineering

• Strong hands-on experience in data engineering, data platform development, or data infrastructure.
• Data ingestion, ETL/ELT, data transformation, and pipeline orchestration.
• Batch processing and/or streaming data processing.
• Experience with distributed data processing or messaging technologies.
• Production experience in at least one relevant domain, such as data pipelines, data warehouses, data lakes, distributed systems, operational databases, Kubernetes, or cloud infrastructure.

Programming and Software Development

• Strong proficiency in Python or another general-purpose programming language.
• Experience writing, testing, debugging, reviewing, and maintaining production-quality code.
• SQL and data processing fundamentals.
• Automation, version control, code quality, and CI/CD practices.

Cloud and Data Platforms

• Experience with cloud infrastructure, preferably AWS, or equivalent cloud platforms.
• Experience with one or more relevant data technologies such as Snowflake, Apache Kafka, Apache Spark, PySpark, PostgreSQL, or data lake technologies.
• Understanding of distributed systems, scalability, fault tolerance, and system design.

Production Engineering

• Experience operating production systems and resolving technical incidents.
• Knowledge of monitoring, observability, logging, deployment safety, and recovery mechanisms.
• Understanding of performance optimization, resource utilization, and infrastructure cost management.