Data Engineer

Celebal Technologies; Hyderabad; On-site; INR 1,200,000 - 1,900,000; Full time

Job summary

Celebal Technologies in Hyderabad/Bangalore is seeking a skilled Data Engineer to design and build scalable data pipelines using modern big data technologies. You will work with streaming and batch processing, leveraging Databricks and Delta Lake to ensure reliable storage and fast data access.

Responsibilities include ingesting data from Kafka and various sources, developing and optimizing PySpark pipelines, and coordinating with analysts and business teams to meet data requirements.

Qualifications

• Hands-on with streaming and batch processing.
• Strong PySpark / Spark experience.
• Databricks and Delta Lake expertise.

Responsibilities

• Design and build end-to-end ETL/ELT pipelines.
• Ingest data from Kafka, databases, APIs, etc.

Skills

PySpark

Apache Spark

Databricks

Delta Lake

Apache Kafka

SQL

Data Pipelines

Data Warehousing

Tools

Airflow

Databricks Autoloader

Jenkins

Job description

Company: Celebal Technologies

Location: Hyderabad/Bangalore

Experience: 3-10 Years

Role Overview

We are looking for a skilled Data Engineer with strong experience in building scalable data pipelines using modern big data technologies. The ideal candidate should have hands-on experience with streaming + batch processing, along with deep knowledge of Databricks and Delta Lake.

Key Responsibilities

• Design and build end-to-end ETL/ELT pipelines
• Ingest data from multiple sources like Kafka, databases, APIs, etc.
• Develop and optimize data pipelines using PySpark / Spark
• Work on both batch and real-time (streaming) data processing
• Implement Delta Lake architecture for reliable and scalable data storage
• Handle large-scale data (100GB1TB+) efficiently
• Perform data transformation, cleansing, and validation
• Optimize performance using partitioning, caching, and file formats
• Work with orchestration tools like Airflow for scheduling workflows
• Collaborate with Data Analysts and Business teams for requirements

Required Skills (Must Have)

• Strong experience in PySpark / Apache Spark
• Hands-on with Databricks
• Good understanding of Delta Lake
• Experience with Apache Kafka
• Strong SQL skills (joins, CTEs, window functions, recursion basics)
• Experience in designing scalable data pipelines
• Understanding of data warehousing concepts

Good to Have

• Experience with Microsoft Azure
• Knowledge of Apache Airflow
• Familiarity with Databricks Autoloader
• Experience with CI/CD tools like Jenkins
• Exposure to data modeling concepts

What Were Looking For

• Someone who has worked on real production pipelines
• Strong problem-solving mindset
• Ability to handle large-scale data efficiently
• Clear understanding of streaming + batch architecture