Sr. Software Engineer Database Level 1

Bebo Technologies; Chandigarh; On-site; INR 1,800,000 - 2,400,000; Full time

Job summary

Bebo Technologies is seeking a senior Data Engineer to design and implement scalable data platforms and pipelines. You will work with Data Lake/Lakehouse/Data Mesh patterns, building batch and real-time pipelines using Databricks, Spark, dbt and AWS services.

You will develop Python microservices and APIs to enable secure data sharing, while ensuring data quality and observability across complex data workflows.

Qualifications

• 6-8 years of experience in Data Engineering or related roles.
• Strong understanding of modern data architectures including Data Lake, Lakehouse and Data Mesh.
• Strong proficiency in Python, PySpark, and SQL.
• Hands-on experience with Databricks, Delta Lake and Snowflake.
• Experience with ETL/ELT, data orchestration tools, and Airflow or Databricks Workflows.
• Experience with streaming tech (Kafka, AWS Kinesis) and batch processing (Spark, Glue, dbt).
• Proficiency with structured, semi-structured, and unstructured data and data modeling.
• Experience with IaC (Terraform) and containerization (Docker).
• Familiarity with data quality, observability, and security concepts.

Responsibilities

• Design and implement scalable data platforms using Data Lake, Lakehouse, and Data Mesh patterns.
• Build and optimize batch and real-time data pipelines with Databricks, Spark, dbt and AWS services.
• Develop robust data ingestion for diverse data sources including APIs and files.
• Develop Python-based microservices for secure data sharing across systems.
• Build RESTful APIs and event-driven services exposing curated datasets.
• Work with Python, PySpark, and SQL for data transformation and workflows.
• Implement streaming pipelines with Kafka or Kinesis and integrate with analytics platforms.
• Design at-scale datasets using Parquet, JSON, CSV formats.
• Optimize storage, partitioning and query performance for analytics workloads.
• Orchestrate data workflows using Airflow and Databricks Workflows.
• Implement IaC and Terraform, containerize apps with Docker.
• Collaborate with data architects, analysts and BI teams to operationalize Lakehouse solutions.
• Contribute to data modeling in Lakehouse environments (Medallion, 3NF).
• Implement data quality, monitoring and observability across pipelines.

Skills

Python

PySpark

SQL

Data engineering

Data lakehouse

Data mesh

ETL/ELT

CI/CD practices

Cloud security concepts

Tools

Databricks

Delta Lake

Snowflake

Airflow

dbt

Kafka

AWS Kinesis

Terraform

Docker

Job description

Key Responsibilities

• Design and implement scalable data platforms leveraging Data Lake, Lakehouse, Data Mesh, and modern data architecture patterns.
• Build and optimize data pipelines for batch and real-time processing using Databricks, Apache Spark, dbt, and cloud-native AWS services.
• Develop robust data ingestion frameworks for structured, semi-structured, and unstructured data from APIs, files, databases, and streaming sources.
• Design and develop scalable Python-based microservices to enable secure and efficient data sharing across systems and applications.
• Build RESTful APIs and event-driven services for exposing curated datasets from Data Lake and Lakehouse platforms.
• Work extensively with Python, PySpark, and SQL for data transformation, processing, and data engineering workflows.
• Implement streaming data pipelines using Kafka/Kinesis and integrate them with downstream analytics and data platforms.
• Design and manage large-scale datasets using formats such as Parquet, JSON, CSV, and IoT/sensor data.
• Optimize data storage, partitioning, and query performance for high-volume analytical workloads.
• Design and implement data orchestration workflows using Apache Airflow and/or Databricks Workflows.
• Implement infrastructure and data platform components using Terraform and Infrastructure as Code (IaC) practices.
• Containerize data applications and services using Docker where applicable.
• Collaborate with cross-functional teams, including Data Architects, Data Analysts, and BI teams, to operationalize Data Lake and Lakehouse solutions.
• Contribute to data modeling in Lakehouse environments, including Medallion Architecture and dimensional modeling.
• Implement data quality, validation, monitoring, and observability practices across data pipelines.
• Use data quality and observability tools to identify data issues, monitor pipeline health, and improve data reliability.
• Ensure data quality, reliability, security, and observability across data pipelines.
• Implement serverless data processing and pipeline solutions where applicable.

Required Skills & Experience

• 6-8 years of experience in Data Engineering or related roles.
• Strong understanding of modern data architectures such as Data Lake, Lakehouse, Data Mesh, and Data Products.
• Strong proficiency in Python, PySpark, and SQL.
• Hands-on experience with Databricks, Delta Lake, and/or Snowflake.
• Experience with ETL/ELT frameworks and data orchestration tools.
• Hands-on experience with Apache Airflow and/or Databricks Workflows.
• Practical experience with streaming technologies such as Kafka and/or AWS Kinesis.
• Strong understanding of batch processing frameworks and technologies such as Apache Spark, AWS Glue, and dbt.
• Proficiency in handling structured, semi-structured, and unstructured data.
• Experience with modern data modeling techniques, including Star Schema, Snowflake Schema, and 3NF.
• Familiarity with vector databases and data architectures supporting AI/ML use cases.
• Hands-on experience with AWS services including S3, Glue, Glue Data Catalog, Athena, and Redshift.
• Experience with Infrastructure as Code (IaC) tools such as Terraform.
• Experience with containerization technologies such as Docker.
• Experience with Git, CI/CD, and DevOps practices.
• Experience implementing data quality, monitoring, and observability solutions using relevant tools and frameworks.
• Good understanding of cloud security concepts, including IAM, encryption, access control, and data governance.
• Strong understanding of Data Lakehouse concepts, architecture, and implementation patterns.
• Familiarity with serverless architectures and Python-based serverless data processing solutions.