Aws Data Engineer

Bridgestone - Global Capability Center; Bengaluru; Hybrid; INR 2,500,000 - 4,000,000; Full time

Job summary

Bridgestone - Global Capability Center in Bengaluru is seeking a Data Engineer to design, develop, and maintain scalable data pipelines using AWS, Databricks, Spark, and PySpark.

You will implement Medallion-like layers, optimize data processing, and ensure quality checks, while collaborating with cross-functional teams in an Agile environment.

The role emphasizes production-ready code, CI/CD, troubleshooting production issues, and driving data modernization initiatives.

Qualifications

• 5+ years of experience in Data Engineering, ETL/ELT, and building large-scale data pipelines, with leadership responsibilities where applicable.
• Strong hands-on experience with AWS services including S3, Glue, Aurora/RDS, Lambda, and Step Functions, and cloud-native data practices.
• 2+ years of hands-on experience with Databricks, Spark, and PySpark, including performance optimization and efficient job design.
• Strong SQL and Python/PySpark skills with data modeling, schema mapping, and ETL/ELT transformations.
• Experience designing and optimizing data pipelines and orchestration workflows (scheduling, retries, monitoring, failure recovery).
• Familiarity with software engineering practices: coding standards, code reviews, CI/CD, Git, and automation.
• Experience with data quality, validation, reconciliation, monitoring, and root-cause analysis.
• Understanding of REST APIs and integration with third-party data sources.
• Documentation across delivery lifecycle and cross-functional collaboration in Agile teams.
• Strong communication and collaboration skills across Product, Business, Architecture, QA, API and Engineering teams.
• Solid understanding of Agile practices and iterative delivery.

Responsibilities

• Design, develop, and maintain scalable data engineering solutions using AWS, Databricks, Spark, and PySpark.
• Build robust data pipelines following layered architectures (Raw, Silver, Gold) covering ingestion, transformation, validation, and reconciliation.
• Develop and optimize data solutions with AWS services (S3, Glue, Aurora/RDS, Lambda, Step Functions) and Databricks/Spark.
• Implement orchestration, scheduling, retries, monitoring, and failure recovery for data workflows; troubleshoot production issues and drive root cause analysis.
• Write production-ready code, participate in reviews, testing, CI/CD, deployment, automation, and continuous improvement.
• Support data migration and modernization initiatives across AWS and Databricks; ensure production readiness.
• Collaborate with Product, Business, Architecture, QA, API, and Engineering in an Agile environment; contribute to design, estimation, and planning.
• Provide technical guidance and mentorship; promote reusable frameworks and engineering standards.

Skills

ETL/ELT pipelines

Python/PySpark

SQL

REST APIs

CI/CD

AWS

Databricks

Spark

Tools

AWS

Databricks

Spark

PySpark

Job description

Data Engineer

OVERVIEW OF ROLE

• Design, develop, and maintain scalable, high-performance data engineering solutions using AWS, Databricks, Spark, and PySpark, leveraging strong technical acumen to solve complex data and engineering challenges.
• Build robust data pipelines following Medallion Architecture (Raw, Silver, and Gold layers), covering data ingestion, transformation, processing, data modeling, validation, quality checks, and reconciliation.
• Develop and optimize data solutions using AWS services including S3, Glue, Aurora/RDS, Lambda, and Step Functions, along with Databricks, Spark, and PySpark to improve performance, scalability, reliability, and cloud cost efficiency.
• Implement reliable orchestration and operational workflows covering scheduling, dependencies, retries, error handling, monitoring, and failure recovery, while troubleshooting complex data and production issues and driving root-cause resolution.
• Develop reusable, maintainable, and production-ready code following engineering standards and best practices; contribute to code reviews, testing, CI/CD, deployment, automation, and continuous improvement.
• Support data migration and modernization initiatives across AWS and Databricks, including legacy platform migrations, source-to-target mapping, data validation, reconciliation, and production readiness.
• Collaborate with Product, Business, Architecture, QA, API, and Engineering teams in a cross-functional Agile environment, contributing to technical design discussions, estimation, sprint planning, backlog refinement, and delivery.
• Provide technical guidance and mentorship to other engineers as applicable, promote reusable frameworks and engineering standards, and contribute to resolving complex technical challenges.

REQUIRED QUALIFICATIONS

• 5+ years of experience in Data Engineering, ETL/ELT, and developing large-scale data pipelines, with experience providing technical guidance or leadership as applicable.
• Strong hands-on experience with AWS services including S3, Glue, Aurora/RDS, Lambda, and Step Functions, with a strong understanding of cloud-native data engineering practices.
• 2+ years of hands-on experience with Databricks, Spark, and PySpark, including Spark performance optimization, partitioning, joins, caching, file formats, data skew, and efficient job design.
• Strong SQL and Python/PySpark skills, with experience in data modeling, database design, schema mapping, ETL/ELT, and source-to-target transformations.
• Experience designing and optimizing data pipelines and orchestration workflows, including scheduling, dependencies, retries, error handling, monitoring, failure recovery, and performance optimization.
• Strong understanding of software engineering practices including coding standards, code reviews, testing, Git, CI/CD, Azure DevOps, reusable frameworks, and automation.
• Experience with data quality, validation, reconciliation, monitoring, troubleshooting, and root-cause analysis.
• Understanding of REST APIs, API request/response flows, and integration with APIs and third-party source systems.
• Experience creating, reviewing, and maintaining functional and technical documentation throughout the delivery lifecycle.
• Strong communication and collaboration skills, with experience working across Product, Business, Architecture, QA, API, and Engineering teams in Agile environments.
• Strong understanding of Agile practices, including sprint planning, backlog refinement, estimation, iterative delivery, and production support.

PREFERRED QUALIFICATIONS

• Experience with API development and integration, including hands-on experience working with API-driven data solutions.
• Experience with Redis or other in-memory databases and caching technologies.
• Experience with Databricks and advanced Spark performance optimization techniques.
• Experience developing reusable frameworks, automation, and engineering standards.
• Relevant AWS certifications in Cloud, Data Engineering, or Solutions Architecture.
• Experience developing, debugging, and supporting data solutions within large, cross-functional engineering teams.
• Strong analytical and problem-solving skills, with the ability to manage multiple priorities in a deadlinedriven environment.
• Strong attention to detail and commitment to data quality, reliability, and engineering excellence.