Walk-in | Data Engineer

Orcapod Consulting Services; Chennai; On-site; INR 1,200,000 - 1,500,000; Full time

Job summary

Orcapod Consulting Services in Chennai is seeking an experienced Data Engineer with strong Databricks, PySpark, Python, and SQL skills to design, develop, and maintain scalable data engineering solutions.

The role focuses on building ETL/ELT pipelines, processing large datasets, and implementing data processing using Apache Spark and Delta Lake, with emphasis on data quality and governance.

Qualifications

• Strong hands-on experience with Databricks.
• Proficiency in PySpark, Python, and SQL.
• Experience with Apache Spark and distributed data processing.
• Strong knowledge of ETL/ELT development and data warehousing concepts.
• Experience with Delta Lake and Delta Tables.
• Knowledge of data modeling and performance optimization.
• Experience with cloud platforms such as Azure, AWS, or GCP.
• Understanding of Git, CI/CD, and version control.

Responsibilities

• Design, develop, and maintain scalable data pipelines using Databricks and PySpark.
• Build and optimize ETL/ELT workflows for structured and unstructured data.
• Develop data transformation logic using Python, PySpark, and SQL.
• Work with Delta Lake, Delta Tables, and Lakehouse architecture.
• Implement and manage workflows using Databricks Workflows.
• Integrate data from multiple sources, including databases, APIs, and cloud storage.
• Optimize Spark jobs, cluster performance, and SQL queries.
• Implement data quality checks, error handling, and monitoring.
• Collaborate with data architects, analysts, and business stakeholders.
• Follow data security, governance, and access-control best practices.
• Troubleshoot pipeline failures and ensure reliable data delivery.

Skills

Databricks

PySpark

Python

SQL

Apache Spark

ETL/ELT development

Data warehousing concepts

Data modeling

Git & CI/CD

Tools

Databricks

Delta Lake

Delta Tables

Azure

AWS

GCP

Job description

Job Title: Data Engineer Databricks
Role: Data Engineer

Job Summary

We are looking for an experienced Data Engineer with strong expertise in Databricks, PySpark, Python, and SQL to design, develop, and maintain scalable data engineering solutions. The ideal candidate will have experience building ETL/ELT pipelines, working with large datasets, and implementing data processing solutions using Apache Spark and Delta Lake.

Key Responsibilities

• Design, develop, and maintain scalable data pipelines using Databricks and PySpark.
• Build and optimize ETL/ELT workflows for structured and unstructured data.
• Develop data transformation logic using Python, PySpark, and SQL.
• Work with Delta Lake, Delta Tables, and Lakehouse architecture.
• Implement and manage workflows using Databricks Workflows.
• Integrate data from multiple sources, including databases, APIs, and cloud storage.
• Optimize Spark jobs, cluster performance, and SQL queries.
• Implement data quality checks, error handling, and monitoring.
• Collaborate with data architects, analysts, and business stakeholders.
• Follow data security, governance, and access-control best practices.
• Troubleshoot pipeline failures and ensure reliable data delivery.

Required Skills

• Strong hands-on experience with Databricks.
• Proficiency in PySpark, Python, and SQL.
• Experience with Apache Spark and distributed data processing.
• Strong knowledge of ETL/ELT development and data warehousing concepts.
• Experience with Delta Lake and Delta Tables.
• Knowledge of data modeling and performance optimization.
• Experience with cloud platforms such as Azure, AWS, or GCP.
• Understanding of Git, CI/CD, and version control.