Data Engineer and Senior Data Engineer

Celebal Technologies; Mumbai, Navi Mumbai; On-site; INR 1,200,000 - 1,800,000; Full time

Job summary

Celebal Technologies is seeking a skilled data engineer to design and implement scalable data pipelines on Databricks using PySpark and SQL. You will optimize ETL/ELT workflows and work with Delta Lake for reliable storage, while collaborating with data architects, BI teams and stakeholders to ensure high data quality and governance.

The role emphasizes CI/CD, version control and DevOps practices, with exposure to cloud platforms such as Azure, AWS, or GCP.

Qualifications

• Experience designing and building scalable data pipelines.
• Hands-on with Databricks, PySpark, and SQL for data transformation.
• Implement and optimize Delta Lake storage and governance.
• Develop ETL/ELT workflows for large-volume data processing.
• Collaborate with architects, BI teams, and stakeholders.
• Follow CI/CD, version control, and deployment best practices.

Responsibilities

• Design and develop scalable data pipelines using Databricks.
• Build and optimize ETL/ELT workflows for large data.
• Develop data transformations with PySpark and SQL.
• Work with Delta Lake for reliable storage.
• Maintain pipelines with Databricks Workflows/Lakeflow.
• Perform data integration from multiple sources.
• Optimize Spark jobs and queries.
• Implement data quality, monitoring, and error handling.
• Collaborate with data architects, BI teams, and stakeholders.
• Follow CI/CD, version control, and deployment practices.

Skills

Databricks

PySpark

SQL

Delta Lake

ETL

Data Warehousing

Data Modelling

Cloud Platforms

Git CI/CD

DevOps Practices

Job description

Key Responsibilities

• Design and develop scalable data pipelines using Databricks and PySpark.
• Build and optimize ETL/ELT workflows for large-volume data processing.
• Develop data transformations using PySpark and SQL.
• Work extensively with Delta Lake for reliable and scalable data storage.
• Develop and maintain data pipelines using Databricks Workflows/Lakeflow.
• Perform data integration from multiple structured and unstructured sources.
• Optimize Spark jobs, queries, and pipeline performance.
• Implement data quality, validation, monitoring, and error-handling mechanisms.
• Collaborate with Data Architects, BI teams, and business stakeholders.
• Follow best practices for CI/CD, version control, and deployment.

Required Skills

• Strong hands-on experience with Databricks
• Strong PySpark and SQL skills
• Good understanding of Delta Lake
• Strong experience in Data Engineering and ETL
• Understanding of Data Warehousing and Data Modelling
• Experience with cloud platforms such as Azure / AWS / GCP
• Knowledge of Git, CI/CD and DevOps
• Good analytical and problem-solving skills
• Strong communication and stakeholder management skills

Good to Have

• Experience with Unity Catalog
• Databricks certifications
• Experience with Kafka / Azure Data Factory / Event Hubs
• Knowledge of Lakehouse architecture
• Experience in migrating legacy data platforms to Databricks