Architect - Data Engineering

Quantum Integrators; Bengaluru; Hybrid; INR 4,500,000 - 7,500,000; Full time

Job summary

Quantum Integrators in Bengaluru is seeking an experienced Architect - Data Engineering with 14+ years of experience to lead the design, development, and optimization of scalable data platforms and pipelines.

You will provide technical leadership to a team of data engineers, collaborating with business and technology stakeholders to deliver secure, reliable, and high-performing data solutions using AWS, Databricks, PySpark and Python.

Qualifications

• Experience with Databricks for data engineering and analytics.
• Strong expertise in AWS Cloud Services (S3, Glue, EMR, Lambda, Redshift, IAM, EC2).
• Hands-on experience in designing and developing ETL/ELT pipelines.
• Advanced SQL and Data Modeling skills.
• Strong programming experience in Python.

Responsibilities

• Lead the design, development, and optimization of scalable data platforms and pipelines.
• Provide technical leadership to a team of data engineers.
• Collaborate with business and technology stakeholders to deliver secure, reliable data solutions.
• Define architecture standards and best practices for data platforms.

Skills

Databricks

PySpark

SQL & Performance Optimization

Python

Tools

Docker

Kubernetes

Terraform

CloudFormation

Snowflake

Azure Data Factory

GCP Data Services

Job description

Mode: Hybrid

Experience: Architect: 14+ Years

Job Description:

Role Summary

We are looking for an experienced Architect - Data Engineering with 14+ years of experience and strong expertise in AWS, Databricks, PySpark, and Python. The ideal candidate will lead the design, development, and optimization of scalable data platforms and pipelines, provide technical leadership to a team of data engineers, and collaborate with business and technology stakeholders to deliver secure, reliable, and high-performing data solutions.

Key Skills

• Databricks
• PySpark
• SQL & Performance Optimization
• Python

Must-Have Skills

• Experience with Databricks for data engineering and analytics
• Strong expertise in AWS Cloud Services (S3, Glue, EMR, Lambda, Redshift, IAM, EC2)
• Hands-on experience in designing and developing ETL/ELT pipelines
• Advanced SQL and Data Modeling skills
• Strong programming experience in Python

Good-to-Have Skills

• Big Data technologies such as Spark Streaming, Hadoop, and Kafka
• CI/CD and DevOps practices
• Infrastructure as Code (Terraform, CloudFormation)
• Docker and Kubernetes
• Exposure to Snowflake, Azure Data Factory, or GCP Data Services
• Lead the design, development, and implementation of scalable data engineering solutions
• Define architecture standards and best practices for data platforms
• Review technical designs and ensure adherence to engineering standards
• Drive technical innovation and continuous improvement initiatives
• Mentor and guide a team of Data Engineers
• Foster a collaborative and high-performance engineering culture
• Build and maintain data pipelines using AWS, Databricks, PySpark, and Airflow
• Develop high-performance ETL/ELT processes for structured and unstructured data
• Design and optimize data models to support reporting, analytics, and machine learning use cases
• Ensure high availability, reliability, and scalability of data platforms