Sr Data Engineer (AWS | Multi-Cloud | Databricks)

Cloudthat

Bengaluru

On-site

INR 1,400,000 - 2,800,000

Full time

10 days ago

Get a reply from this employer

Your resume and cover letter, tailored to exactly what they’re hiring for.

Get past ATS filters

Job summary

CloudThat Technologies Pvt. Ltd. seeks a Senior Data Engineer with 5+ years of hands-on experience to design, develop, and implement scalable data engineering solutions across cloud platforms in Bengaluru.

You will work with AWS as the primary cloud, alongside GCP/Azure exposure, Databricks, and Spark to build robust data pipelines and lakehouse architectures. The role focuses on ETL/ELT development, data quality, and performance optimization, with emphasis on scalable, cost-efficient platforms

Qualifications

  • 5+ years of Data Engineering experience.
  • Strong hands-on with AWS data engineering services.
  • Hands-on Databricks and Apache Spark experience.
  • Proficient in Python, PySpark, and SQL.
  • Exposure to GCP or Azure data platforms.
  • Experience designing end-to-end data pipelines.
  • Knowledge of data lakehouse architectures and distributed processing.

Responsibilities

  • Design, develop, and maintain scalable data pipelines using AWS and Databricks.
  • Build batch and real-time data processing with Spark, PySpark, and SQL.
  • Implement data lake/lakehouse architectures (S3, Delta Lake, Iceberg).
  • Work with AWS data services for ingestion, transformation, storage, and analytics.
  • Develop ETL/ELT workflows from multiple data sources.
  • Implement data quality, validation, and reconciliation frameworks.
  • Optimize Spark jobs, queries, and distributed workloads.
  • Orchestrate workflows with Airflow, Step Functions, or Databricks Workflows.
  • Handle streaming data and real-time ingestion/processing.
  • Create reusable frameworks, automation scripts, and data engineering components.
  • Ensure security, access control, encryption, monitoring, and governance.

Skills

AWS data engineering services
Databricks
Apache Spark
PySpark
SQL
Python
GCP
Azure
Data pipelines
Airflow
Databricks Workflows
AWS Step Functions
Delta Lake
Apache Iceberg
S3
Terraform
CloudFormation

Tools

Databricks
Apache Spark
Python
PySpark
SQL
Delta Lake
Apache Iceberg
Airflow
AWS Step Functions
Databricks Workflows
Terraform
CloudFormation

Job description

5+ Yrs

Full-Time

Experience: 5+ Years

Primary Cloud Platform: AWS

Secondary Cloud Platforms: GCP and Microsoft Azure

Core Technology: Databricks, Apache Spark, PySpark, SQL, Data Engineering

Job Overview

We are looking for an experienced Senior Data Engineer with 5+ years of hands-on experience in designing, developing, and implementing scalable data engineering solutions across cloud platforms.

The ideal candidate should have strong expertise in AWS as the primary cloud platform, along with working knowledge and exposure to data engineering services across Google Cloud Platform (GCP) and Microsoft Azure.

The candidate must possess excellent data engineering fundamentals, strong programming and problem-solving skills, and very good hands-on experience with Databricks and Apache Spark.

The role involves building modern data pipelines, implementing data lakes and lakehouse architectures, optimizing large-scale data processing workloads, and developing reliable, scalable, and cost-efficient data platforms.

Key Responsibilities
  • Design, develop, and maintain scalable end-to-end data pipelines using AWS and Databricks.
  • Build and optimize batch and real-time data processing solutions using Apache Spark, PySpark, and SQL.
  • Design and implement modern data lake and lakehouse architectures using S3, Delta Lake, Apache Iceberg, and Databricks.
  • Work with AWS data engineering services for data ingestion, transformation, orchestration, storage, cataloging, and analytics.
  • Develop ETL/ELT workflows integrating structured, semi-structured, and unstructured data from multiple sources.
  • Implement data transformation, data quality, validation, and reconciliation frameworks.
  • Optimize Spark jobs, SQL queries, cluster configurations, and distributed processing workloads.
  • Implement workflow orchestration using Airflow, AWS Step Functions, Databricks Workflows, or equivalent services.
  • Work with streaming technologies and services for real-time data ingestion and processing.
  • Develop reusable frameworks, automation scripts, and standardized data engineering components.
  • Implement security, access control, encryption, monitoring, logging, and governance across data platforms.
  • Collaborate with architects, analysts, data scientists, and business stakeholders to translate requirements into scalable technical solutions.
  • Participate in architecture discussions, technical design, code reviews, and performance optimization initiatives.
Required Experience & Qualifications
  • 5+ years of experience in Data Engineering.
  • Strong hands-on experience with AWS data engineering services.
  • Very good practical experience with Databricks and Apache Spark.
  • Strong programming skills in Python, PySpark, and SQL.
  • Practical exposure to at least one of GCP or Azure, with familiarity with the data engineering ecosystem of both.
  • Experience designing and implementing end-to-end data pipelines.
  • Understanding of modern data lakehouse architecture and distributed data processing.
  • Experience working with large-scale datasets and performance optimization.
Good to Have
  • Experience working on multi-cloud data engineering architectures.
  • Exposure to AWS, Azure, and GCP migrations.
  • Experience with Apache Kafka, Amazon MSK, or other streaming platforms.
  • Knowledge of Apache Iceberg, Delta Lake, and open table formats.
  • Exposure to data governance, lineage, and metadata management.
  • Knowledge of DevOps practices and CI/CD automation.
  • Experience with Infrastructure as Code (Terraform / CloudFormation).
  • Exposure to Generative AI data pipelines and vector databases.
Ideal Candidate Profile

The ideal candidate should be a hands-on Data Engineer who can independently design, develop, troubleshoot, and optimize complex data engineering pipelines.

We are specifically looking for someone who:

  • Has strong practical expertise in AWS data engineering services.
  • Has very good hands-on experience with Databricks and PySpark.
  • Understands data engineering ecosystems across AWS, GCP, and Azure.
  • Has strong fundamentals in distributed data processing, SQL, Python, and data architecture.
  • Can work independently on end-to-end data engineering implementations.
  • Can contribute to technical architecture and engineering best practices.
Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Sr. Databricks Engineer
Sr. Databricks Engineer

Fa Ewjt Saasfaprod1 • Pune District

Hybrid
INR 3,500,000 - 6,000,000
Sr. Databricks Engineer
Sr. Databricks Engineer

ExlService Holdings, Inc. • Bengaluru

Hybrid
INR 4,500,000 - 7,500,000
Data Engineer ( AWS & Databricks)
Data Engineer ( AWS & Databricks)

Tekskills Inc. • Bengaluru

On-site
INR 1,500,000 - 2,100,000
Data Engineer ( AWS & Databricks)
Data Engineer ( AWS & Databricks)

Tekskills Inc. • Kolkata District

On-site
INR 1,500,000 - 2,300,000
Data Engineer ( AWS & Databricks)
Data Engineer ( AWS & Databricks)

Tekskills Inc. • Dadri

On-site
INR 1,400,000 - 2,500,000
Data Engineer ( AWS & Databricks)
Data Engineer ( AWS & Databricks)

Tekskills Inc. • Mumbai

On-site
INR 1,500,000 - 2,400,000
Databricks Engineer
Databricks Engineer

Impronics Technologies • Gurugram District

On-site
INR 4,000,000 - 7,000,000
Data Engineer ( AWS & Databricks)
Data Engineer ( AWS & Databricks)

Tekskills Inc. • Hyderabad

On-site
INR 1,200,000 - 1,800,000
Senior Data Engineer Databricks / PySpark
Senior Data Engineer Databricks / PySpark

Wissen Technology Pte Ltd • Bengaluru, Mumbai

On-site
INR 1,500,000 - 2,300,000
Senior Data Engineer
Senior Data Engineer

Amgen Technology • Hyderabad

Hybrid
INR 2,500,000 - 4,000,000