Data Engineer

Qcentrio; Hyderabad; On-site; INR 1,800,000 - 3,200,000; Full time

Job summary

Qcentrio, Hyderabad, seeks an experienced Big Data Engineer to design, build, and maintain scalable ETL/ELT pipelines for terabytes of data, using Spark, Hadoop, and modern data architectures.

You will optimize data lake/lakehouse implementations, collaborate with data science and software teams, ensure data quality and security, and drive cost-efficient operations in Azure/AWS environments.

Qualifications

• 7–10 years of relevant experience in big data engineering.
• Advanced proficiency in Python and SQL for data manipulation and analysis.
• Hands-on experience with Apache Spark, Hadoop, or similar distributed systems.
• Cloud development experience with Azure, AWS, or GCP.
• Solid understanding of data lake and data lakehouse architectures.
• Expertise in ETL performance tuning and cost optimization techniques.

Responsibilities

• Design, develop, and maintain scalable ETL/ELT pipelines for processing large volumes of data (TBs).
• Model and structure data for performance, scalability, and usability.
• Work with cloud infrastructure (preferably Azure) to build and optimize data workflows.
• Leverage Spark and Hadoop for large-scale data processing.
• Build and manage data lake/lakehouse architectures per best practices.
• Optimize ETL performance and cost-efficient data operations.
• Collaborate with data science, analytics, and software engineering teams.
• Ensure data quality, integrity, and security across all stages of the data lifecycle.

Skills

Python

SQL

AWS

Azure

Hadoop

Education

Bachelor's degree in CS/EE/IS or related field

Tools

Apache Airflow

Azure Data Factory

Job description

Key Responsibilities :

• Design, develop, and maintain scalable and reliable ETL/ELT pipelines for processing large volumes of data (terabytes and beyond).


• Model and structure data for performance, scalability, and usability.


• Work with cloud infrastructure (preferably Azure) to build and optimize data workflows.


• Leverage distributed computing frameworks like Apache Spark and Hadoop for large-scale data processing.


• Build and manage data lake/lakehouse architectures in alignment with best practices.


• Optimize ETL performance and manage cost-effective data operations.


• Collaborate closely with cross-functional teams including data science, analytics, and software engineering.


• Ensure data quality, integrity, and security across all stages of the data lifecycle.

Required Skills & Qualifications :

• 7 to 10 years of relevant experience in bigdata engineering.


• Advanced proficiency in Python,


• Strong skills in SQL for complex data manipulation and analysis.


• Hands-on experience with Apache Spark, Hadoop, or similar distributed systems.


• Proven track record of handling large-scale datasets (TBs) in production environments.


• Cloud development experience with Azure (preferred), AWS, or GCP.


• Solid understanding of data lake and data lakehouse architectures.


• Expertise in ETL performance tuning and cost optimization techniques.


• Knowledge of data structures, algorithms, and modern software engineering practices.

Soft Skills :

• Strong communication skills with the ability to explain complex technical concepts clearly and concisely.


• Self-starter who learns quickly and takes ownership.


• High attention to detail with a strong sense of data quality and reliability.


• Comfortable working in an agile, fast-changing environment with incomplete requirements.

Preferred Qualifications :

• Experience with tools like Apache Airflow, Azure Data Factory, or similar.


• Familiarity with CI/CD and DevOps in the context of data engineering.


• Knowledge of data governance, cataloging, and access control principles.

Skills : Python,Sql,Aws,Azure, Hadoop