Data Engineer

Kapi Technologies; Bengaluru; On-site; INR 1,500,000 - 2,500,000; Full time

Job summary

Kapi Technologies in Bengaluru is seeking a seasoned Data Engineer to design, develop, and maintain scalable data pipelines. You will build ETL/ELT workflows with DBT and Apache Airflow, optimize SQL queries, and model data for high-performance processing in a modern data platform.

You should have 5+ years of hands-on experience with PySpark, Spark or Flink, Data Lake architectures with Apache Iceberg, and cloud platforms (AWS).

Qualifications

• Minimum 5+ years of experience as a Data Engineer or in a similar role.
• Hands-on experience with DBT, Apache Airflow, PySpark, and SQL.
• Experience with Data Lake solutions using Apache Iceberg.
• Strong understanding of ETL/ELT and modern data architectures.
• Proficiency in Python for data engineering and automation is preferred.
• Experience with Spark or Flink for distributed processing.
• Working knowledge of AWS cloud services (S3, Athena, MWAA, EKS).
• Familiarity with Kubernetes, Parquet, ORC, and Iceberg formats is a plus.
• Good CI/CD, version control, testing, monitoring, and DevOps practices.
• Excellent analytical, problem-solving, and communication skills.

Responsibilities

• Design, develop, and maintain scalable, high-performance data pipelines.
• Build and manage ETL/ELT workflows using DBT and Airflow.
• Develop and optimize complex SQL queries for large-scale processing.
• Design robust data models ensuring quality and consistency.
• Develop data processing with PySpark, Spark, or Flink.
• Implement Data Lake architectures using Apache Iceberg.
• Monitor database performance, storage, and query efficiency.
• Collaborate with product managers, data scientists, analysts, and engineers.
• Implement CI/CD, testing, monitoring, and documentation best practices.
• Troubleshoot production issues to ensure reliability, scalability, and security.

Skills

DBT

Apache Airflow

PySpark

SQL

Python

Data Warehousing

CI/CD

DevOps

Agile/Scrum

Communication

Education

Bachelor's or Master's degree in Computer Science, IT, Engineering, or related field

Tools

Apache Iceberg

DBT

Apache Airflow

PySpark

Apache Spark

Apache Flink

Kubernetes

Parquet

ORC

Job description

Roles & Responsibilities

• Design, develop, and maintain scalable, reliable, and high-performance data pipelines.
• Build and manage ETL/ELT workflows using DBT and Apache Airflow.
• Develop and optimize complex SQL queries for large-scale data processing.
• Design and implement robust data models while ensuring data quality and consistency.
• Develop data processing solutions using PySpark, Apache Spark, or Apache Flink.
• Design, implement, and manage Data Lake architectures using Apache Iceberg.
• Monitor and optimize database performance, storage, and query efficiency.
• Collaborate with Product Managers, Data Scientists, Analysts, and Engineering teams to deliver business-driven data solutions.
• Implement CI/CD pipelines, testing, monitoring, logging, and documentation best practices.
• Troubleshoot production issues and ensure the reliability, scalability, and security of data platforms.
• Continuously improve data engineering processes, architecture, and performance.

Preferred candidate profiles

• Minimum 5+ years of experience as a Data Engineer or in a similar data engineering role.
• Strong hands-on experience with DBT, Apache Airflow, PySpark, and SQL.
• Experience in designing and implementing Data Lake solutions using Apache Iceberg.
• Strong understanding of ETL/ELT, Data Warehousing, and modern data architectures.
• Proficiency in Python for data engineering and automation (Preferred).
• Experience with Apache Spark or Apache Flink for distributed data processing.
• Working knowledge of cloud platforms, preferably AWS (S3, Athena, MWAA, EKS, ElastiCache, SageMaker).
• Familiarity with Kubernetes, YARN, Parquet, ORC, and Iceberg file formats is an added advantage.
• Good understanding of CI/CD, version control, testing, monitoring, and DevOps practices.
• Strong analytical, problem-solving, and debugging skills.
• Excellent communication and stakeholder management abilities.
• Experience working in Agile/Scrum development environments.
• Bachelor's or Master's degree in Computer Science, Information Technology, Engineering, or a related field.
• Candidates with experience building scalable, secure, and high-performance data platforms will be preferred.