Associate-Data Engineer

Acuity Knowledge Partners; Bengaluru; On-site; INR 900,000 - 1,400,000; Full time

Job summary

Acuity Knowledge Partners in Bengaluru seeks a Data Engineer to join a multi-year transformation program. You will design, develop and implement cloud-native data platforms, building scalable pipelines with Databricks, PySpark and streaming frameworks.

Collaborate with architects, data engineers and stakeholders to deliver real-time and batch data solutions across AWS and Azure. The role emphasizes hands-on Databricks development, streaming with Kafka, and data lake management, plus strong SQL

Qualifications

• Bachelor's degree in Computer Science, Engineering, Information Systems or a related field.
• 2–5 years of experience in Data Engineering or related technical roles.
• Hands-on experience with Databricks development and administration.
• Experience developing data pipelines using PySpark and Spark SQL.
• Knowledge of cloud platforms including AWS and/or Microsoft Azure.
• Experience working with data lakes, distributed computing and large-scale data processing.
• Understanding of real-time data streaming concepts and architectures, with familiarity with Databricks Structured Streaming.
• Working knowledge of Apache Kafka, AWS MSK or Confluent Kafka.
• Experience with source-control systems such as Git, and strong SQL skills.
• Strong analytical, problem-solving and communication skills.

Responsibilities

• Develop, maintain and optimize data pipelines using Databricks and cloud-native technologies.
• Design and implement batch and real-time data ingestion frameworks.
• Build and support streaming data solutions using Databricks Structured Streaming.
• Integrate data sources with event-streaming platforms such as AWS MSK and Confluent Kafka.
• Develop ETL/ELT processes to transform and load data into enterprise data platforms.
• Participate in data modeling, data-quality validation and metadata management activities.
• Collaborate with business, technology and architecture teams to understand data requirements and translate them into scalable solutions.
• Support cloud migration and modernization initiatives across AWS and Azure environments.
• Monitor, troubleshoot and optimize data pipelines for performance, reliability and scalability.
• Participate in code reviews, testing, deployment and production-support activities.
• Contribute to establishing engineering standards, best practices and reusable frameworks.
• Bachelor's degree in Computer Science, Engineering, Information Systems or a related field.
• 2–5 years of experience in Data Engineering or related technical roles.
• Hands-on experience with Databricks development and administration.
• Experience developing data pipelines using PySpark and Spark SQL.
• Knowledge of cloud platforms including AWS and/or Microsoft Azure.
• Experience working with data lakes, distributed computing and large-scale data processing.
• Understanding of real-time data streaming concepts and architectures, with familiarity with Databricks Structured Streaming.
• Working knowledge of Apache Kafka, AWS MSK or Confluent Kafka.
• Experience with source-control systems such as Git, and strong SQL skills.
• Strong analytical, problem-solving and communication skills.

Skills

Databricks

PySpark

Spark SQL

Kafka

SQL

Git

REST APIs

Airflow

Kubernetes

Docker

CI/CD

Terraform

Education

Bachelor's degree in Computer Science, Engineering, Information Systems or a related field

Tools

AWS

Azure

Databricks

Confluent Kafka

MSK

Git

SQL

Airflow

Docker

Kubernetes

Terraform

Job description

We are seeking a Data Engineer for a multi-year strategic transformation program modernizing the global middle- and back-office technology platform through a cloud-native, event-driven architecture. The platform leverages AWS, Azure, Kafka-based event streaming, Databricks, modern workflow orchestration, API-led integration and scalable cloud infrastructure. We are seeking a highly motivated Analyst-level Data Engineer to support the design, development and implementation of modern cloud-based data platforms. The candidate will work closely with business stakeholders, solution and data architects, data engineers from other teams at SMBC, and application teams to build scalable data pipelines and streaming solutions that support strategic transformation initiatives. The ideal candidate will have hands-on experience with Databricks, cloud technologies (AWS and Azure) and real-time data processing frameworks; experience working with Apache Kafka, AWS MSK or Confluent Kafka is highly desirable.

Key responsibilities

• Develop, maintain and optimize data pipelines using Databricks and cloud-native technologies.
• Design and implement batch and real-time data ingestion frameworks.
• Build and support streaming data solutions using Databricks Structured Streaming.
• Integrate data sources with event-streaming platforms such as AWS MSK (Managed Streaming for Kafka) and Confluent Kafka.
• Develop ETL/ELT processes to transform and load data into enterprise data platforms.
• Participate in data modeling, data-quality validation and metadata management activities.
• Collaborate with business, technology and architecture teams to understand data requirements and translate them into scalable solutions.
• Support cloud migration and modernization initiatives across AWS and Azure environments.
• Monitor, troubleshoot and optimize data pipelines for performance, reliability and scalability.
• Participate in code reviews, testing, deployment and production-support activities.
• Contribute to establishing engineering standards, best practices and reusable frameworks.
• Bachelor's degree in Computer Science, Engineering, Information Systems or a related field.
• 2–5 years of experience in Data Engineering or related technical roles.
• Hands-on experience with Databricks development and administration.
• Experience developing data pipelines using PySpark and Spark SQL.
• Knowledge of cloud platforms including AWS and/or Microsoft Azure.
• Experience working with data lakes, distributed computing and large-scale data processing.
• Understanding of real-time data streaming concepts and architectures, with familiarity with Databricks Structured Streaming.
• Working knowledge of Apache Kafka, AWS MSK or Confluent Kafka.
• Experience with source-control systems such as Git, and strong SQL skills.
• Strong analytical, problem-solving and communication skills.

Preferred qualifications

• Experience with Delta Lake architecture and Lakehouse implementations.
• Knowledge of event-driven architectures and messaging patterns.
• Experience with CI/CD pipelines and DevOps practices.
• Familiarity with Terraform, CloudFormation or other Infrastructure-as-Code frameworks.
• Experience with financial services, capital markets or regulatory reporting platforms.
• Understanding of trade lifecycle concepts across FX, Rates, Derivatives and Securities.
• Exposure to enterprise integration technologies and API-based architectures.
• Additional preferred tooling: Airflow, Docker, Kubernetes, Python, REST APIs and data modeling.