Senior Data Engineer

Orcapod Consulting Services; Chennai District; On-site; INR 1,500,000 - 2,100,000; Full time

Job summary

Orcapod Consulting Services in Chennai seeks a seasoned Data Engineer to design and implement end-to-end data pipelines using Python, PySpark, and SQL. You will collaborate with engineers, researchers and data scientists to gather requirements and deliver scalable solutions on Azure Data Lake and Databricks.

The role requires 5–8+ years of Python experience, strong ETL skills, and familiarity with DevOps practices.

Qualifications

• 5+ years of experience using Python in data engineering
• Experience developing data pipelines with Pandas, NumPy, PySpark and SQL
• Working knowledge of DevOps and Databricks
• Experience with Azure cloud data lake storage and Azure services, and SQL ETL processes

Responsibilities

• Work with engineers, researchers, and data scientists to gather requirements and develop data pipelines
• Independently develop, code, test, and debug data applications of varying complexity
• Conduct experiments and data analyses to ensure robustness, scalability, and accuracy of solutions
• Optimize and fine‑tune pipelines for performance and efficiency

Skills

Python

Pandas

NumPy

PySpark

SQL

DevOps

Databricks

Tools

Azure Data Lake Storage

Azure Databricks

Job description

Job Requirements

• 5 to 8 + Years of experience using Python.
• Develop and implement Data pipelines and algorithms using Python programming language, jointly with knowledge about Pandas, NumPy, Pyspark, SQL and similar libraries.
• Should have working knowledge in devops and should be well versed in databricks to solve real-world problems.
• Should have working knowledge in Azure cloud data lake storage and azure services.
• Utilize SQL for data Extraction, Transformation, and Loading (ETL) processes.

Key Responsibilities

• Work closely with engineers, researchers, and data scientists to gather requirements, perform data analysis, and develop end-to-end pipelines.
• Independently develop, code or modify, test, and debug application programs of varying degree of complexity
• Conduct experiments and perform data analysis to ensure the robustness, scalability, and accuracy of provided solutions
• Optimize and fine-tune for performance and efficiency.

Soft Skills

• Excellent communication
• Team collaboration
• Documentation and knowledge sharing

Certifications

Microsoft Azure Certified Developer / Azure Databricks Data Engineer

Skills: Digital : Python~Digital : PySpark