Big Data Engineer

SP Software

Bengaluru

On-site

INR 2,500,000 - 4,500,000

Full time

4 days ago
Be an early applicant

Get a reply from this employer

Your resume and cover letter, tailored to exactly what they’re hiring for.

Get past ATS filters

Job summary

SP Software is seeking a Senior Big Data Engineer to design and implement scalable data pipelines using PySpark, Scala and Spark, with strong AWS cloud services experience. The role focuses on ETL/ELT development, data processing, and pipeline orchestration, ensuring performance, reliability and data quality across complex systems.

You will collaborate with cross-functional teams, leverage Databricks and Snowflake for data analytics, and apply best practices in monitoring and optimization.

Qualifications

  • Experience designing scalable big data pipelines using PySpark, Scala and Spark.
  • Hands-on expertise with AWS cloud services (EMR, Glue, Step Functions, Lambda).
  • Strong ETL/ELT development skills and data pipeline architecture knowledge.
  • Experience with data processing, orchestration, monitoring and optimization.
  • Familiarity with Git for source control and collaboration.

Responsibilities

  • Design, develop and maintain scalable big data pipelines with PySpark and Spark.
  • Build and optimize ETL/ELT workflows for large data volumes.
  • Develop data processing solutions using EMR and AWS Glue.
  • Implement workflow orchestration with Step Functions and Lambda.
  • Monitor, troubleshoot and optimize pipelines for performance and quality.
  • Collaborate with cross-functional teams and leverage Databricks/Snowflake.
  • Utilize AWS services to create secure, scalable data solutions.
  • Collaborate with teams to understand data requirements and deliver reliable outcomes.
  • Use Git for version control and collaborative development.

Skills

PySpark
Scala
Spark
AWS Cloud
ETL Development
Big Data Pipelines
Pipeline Orchestration
Data Processing
Git

Tools

Databricks
Snowflake
AWS Step Functions
AWS Lambda
Amazon EMR
AWS Glue

Job description

Mandatory Skills

PySpark, Scala, Apache Spark, AWS, Amazon EMR, AWS Glue, AWS Step Functions, AWS Lambda, ETL Development and Big Data Pipeline Development.

Role & responsibilities

We are looking for an experienced Senior Big Data Engineer with strong expertise in PySpark, Scala, Apache Spark and AWS cloud services. The candidate will be responsible for designing, developing, implementing and optimizing scalable big data pipelines and ETL solutions. The role requires hands-on experience with AWS services such as Amazon EMR, AWS Glue, AWS Step Functions and AWS Lambda, along with expertise in data processing, pipeline orchestration, monitoring and performance optimization.

Key Responsibilities
  • Design, develop and maintain scalable big data pipelines using PySpark, Scala and Apache Spark.
  • Build and manage ETL/ELT workflows for processing large volumes of structured and unstructured data.
  • Develop and optimize data processing solutions using Amazon EMR and AWS Glue.
  • Implement workflow orchestration and automation using AWS Step Functions and AWS Lambda.
  • Monitor, troubleshoot and optimize data pipelines to ensure performance, reliability and data quality.
  • Work with Databricks and Snowflake for data processing, integration and analytics.
  • Leverage additional AWS services to build secure, scalable and efficient data engineering solutions.
  • Collaborate with cross-functional teams to understand data requirements and deliver reliable data solutions.
  • Use Git for source code management, version control and collaborative development.
  • Explore and leverage AI-related technologies and capabilities to improve data engineering workflows.
Required Technical Skills
  • Strong hands-on experience in PySpark, Scala and Apache Spark.
  • Extensive experience with AWS cloud services.
  • Practical experience with Amazon EMR and AWS Glue.
  • Experience in AWS Step Functions and AWS Lambda.
  • Strong knowledge of ETL development and big data pipeline architecture.
  • Experience in data pipeline monitoring, troubleshooting and performance optimization.
  • Working knowledge of Git and version control.
Good-to-Have Skills
  • Experience with Databricks and Snowflake.
  • Knowledge of AI / Generative AI concepts and applications.
  • Exposure to additional AWS services such as S3, Redshift, Athena and CloudWatch.
  • Experience with data quality, workflow automation and distributed data processing.
Preferred Skills

Databricks, Snowflake, AI Knowledge, AWS Services, Data Pipeline Monitoring and Performance Optimization.

Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Senior Data Engineer
Senior Data Engineer

Chandigarh News • Mohali, Chandigarh

On-site
INR 900,000 - 1,500,000
Aws Data Engineer
Aws Data Engineer

Qentelli LLC • Bengaluru

On-site
INR 1,400,000 - 2,200,000
AWS Big Data Engineer
AWS Big Data Engineer

Zorba AI • Hyderabad

On-site
INR 1,500,000 - 2,500,000
Data Engineer Lead
Data Engineer Lead

Iris Software, Inc. • Dadri, Pune District, Gurugram District

On-site
INR 900,000 - 1,400,000
Data Engineer
Data Engineer

Bdreamz Global Solutions • Chennai

Hybrid
INR 1,200,000 - 2,200,000
Senior Data Engineer
Senior Data Engineer

AagatiServe Pvt Ltd • Delhi

On-site
INR 1,800,000 - 2,400,000
Data Software Engineer
Data Software Engineer

EPAM Systems Inc • Coimbatore District

On-site
INR 1,800,000 - 2,800,000
Big Data Engineer
Big Data Engineer

HNM Systems • Hyderabad

Hybrid
INR 1,200,000 - 1,800,000
Senior Data Engineer
Senior Data Engineer

CloudBoson • Dadri

On-site
INR 1,800,000 - 3,000,000
Data Engineer - CB2
Data Engineer - CB2

Bridgestone Americas, Inc. • Bengaluru

On-site
INR 1,400,000 - 2,800,000