Aws Data Engineer

Bct Consulting; Bengaluru; On-site; INR 1,400,000 - 2,500,000; Full time

Job summary

Bct Consulting is seeking an AWS Data Engineer in Bengaluru to design and maintain scalable data pipelines. The role requires 4-8 years of hands-on experience with Spark, Python, and SQL, focusing on scalable ETL/ELT processes and data platforms.

You will collaborate with data analysts and stakeholders, ensure data quality, and optimize performance and costs in a cloud-based data landscape. Immediate joiners preferred and Bengaluru-based candidates are encouraged to apply.

Qualifications

• 4-8 years of experience in Data Engineering.
• Hands-on experience with Apache Spark (PySpark/Spark SQL).
• Proficiency in Python for data processing and automation.

Responsibilities

• Design, develop, and optimize scalable ETL/ELT data pipelines using Spark and Python.
• Build and maintain data ingestion frameworks for structured and unstructured data.
• Write optimized SQL queries for data extraction, transformation and reporting.
• Collaborate with data analysts, scientists, and stakeholders to understand data requirements.
• Monitor data pipelines, troubleshoot production issues, and ensure data quality and reliability.
• Implement data validation, logging, and monitoring mechanisms.
• Optimize Spark jobs for performance and cost efficiency.
• Work with large-scale datasets in distributed environments.
• Follow data governance, security, and compliance standards.
• Participate in code reviews and improve engineering practices.

Skills

Apache Spark

Python

SQL

ETL/ELT

Data pipelines

Git

CI/CD

Data quality

Analytical skills

Tools

Airflow

Kafka

Delta Lake

Apache Iceberg

Terraform

CloudFormation

AWS

Job description

AWS Data Engineer

4 - 8 Years of Experience

Location: Bengaluru (Preferred Bengaluru based candidates to apply)

Should be available to attend the Face to Face Interview

Immediate Joiners Preferred

About the Role

We are looking for a skilled Data Engineer with 4 - 8 years of experience in building and maintaining scalable data pipelines and data platforms. The ideal candidate should have hands‑on expertise in AWS, Spark, SQL, Python with a strong understanding of data engineering best practices and cloud‑based data solutions.

Key Responsibilities

• Design, develop, and optimize scalable ETL/ELT data pipelines using Spark and Python.


• Build, maintain, and enhance data ingestion frameworks for structured and unstructured data.


• Write efficient and optimized SQL queries for data extraction, transformation, and reporting.


• Collaborate with data analysts, data scientists, and business stakeholders to understand data requirements.


• Monitor data pipelines, troubleshoot production issues, and ensure data quality and reliability.


• Implement data validation, logging, and monitoring mechanisms.


• Optimize Spark jobs for performance and cost efficiency.


• Work with large‑scale datasets in distributed computing environments.


• Follow data governance, security, and compliance standards.


• Participate in code reviews and contribute to continuous improvement of engineering practices.

Required Skills

• 4 - 8 years of experience in Data Engineering.


• Strong hands‑on experience with Apache Spark (PySpark or Spark SQL).


• Proficiency in Python for data processing and automation.


• Advanced SQL skills with experience in query optimization.


• Experience in building batch and/or streaming data pipelines.


• Knowledge of data modeling and data warehousing concepts.


• Experience with Git and CI/CD practices.


• Strong analytical, problem‑solving, and debugging skills.

Preferred Qualifications

• Experience with Apache Airflow or similar workflow orchestration tools.


• Exposure to Kafka or other streaming technologies.


• Familiarity with Delta Lake, Apache Iceberg, or Apache Hudi.


• Experience with Infrastructure as Code (Terraform or CloudFormation) is a plus.


• AWS Certification is an added advantage.