Big Data Engineer - Python

Qcentrio; Surat; On-site; INR 350,000 - 600,000; Full time

Job summary

Qcentrio is looking for an experienced Data Engineer to design and build scalable ETL pipelines and manage multi-terabyte datasets. The role requires strong Python and SQL skills, plus hands-on experience with Spark, Hadoop, and cloud platforms like Azure or AWS.

You will work across data science, analytics, and software teams to ensure data quality and efficient data workflows. Prior experience with data lakehouse architectures and Airflow/Data Factory is preferred.

Qualifications

• 7 to 10 years of relevant experience in bigdata engineering.
• Advanced proficiency in Python, SQL, and cloud-based data platforms.
• Hands-on experience with distributed systems (Spark, Hadoop).
• Experience managing large-scale datasets (TBs) in production.
• Cloud development experience with Azure, AWS, or GCP.
• Knowledge of data lake/lakehouse architectures and ETL optimization.

Responsibilities

• Design, develop, and maintain scalable ETL/ELT pipelines for large data volumes.
• Model and structure data for performance, scalability, and usability.
• Work with cloud infrastructure (Azure/AWS/GCP) to optimize data workflows.
• Leverage distributed computing frameworks for large-scale processing.
• Build and manage data lake/lakehouse architectures in line with best practices.
• Collaborate with data science, analytics, and software teams.
• Ensure data quality, integrity, and security across the lifecycle.

Skills

Python

SQL

AWS

Azure

Hadoop

Apache Spark

Tools

Apache Airflow

Azure Data Factory

Job description

Job description :

We are seeking an experienced and driven Data Engineer with 5+ years of hands-on experience in building scalable data infrastructure and systems. You will play a key role in designing and developing robust, high-performance ETL pipelines and managing large-scale datasets to support critical business functions. This role requires deep technical expertise, strong problem-solving skills, and the ability to thrive in a fast-paced, evolving environment.

Key Responsibilities :

• Design, develop, and maintain scalable and reliable ETL/ELT pipelines for processing large volumes of data (terabytes and beyond).
• Model and structure data for performance, scalability, and usability.
• Work with cloud infrastructure (preferably Azure) to build and optimize data workflows.
• Leverage distributed computing frameworks like Apache Spark and Hadoop for large-scale data processing.
• Build and manage data lake/lakehouse architectures in alignment with best practices.
• Optimize ETL performance and manage cost-effective data operations.
• Collaborate closely with cross-functional teams including data science, analytics, and software engineering.
• Ensure data quality, integrity, and security across all stages of the data lifecycle.

Required Skills & Qualifications :

• 7 to 10 years of relevant experience in bigdata engineering.
• Advanced proficiency in Python,
• Strong skills in SQL for complex data manipulation and analysis.
• Hands-on experience with Apache Spark, Hadoop, or similar distributed systems.
• Proven track record of handling large-scale datasets (TBs) in production environments.
• Cloud development experience with Azure (preferred), AWS, or GCP.
• Solid understanding of data lake and data lakehouse architectures.
• Expertise in ETL performance tuning and cost optimization techniques.
• Knowledge of data structures, algorithms, and modern software engineering practices.

Soft Skills :

• Strong communication skills with the ability to explain complex technical concepts clearly and concisely.
• Self-starter who learns quickly and takes ownership.
• High attention to detail with a strong sense of data quality and reliability.
• Comfortable working in an agile, fast-changing environment with incomplete requirements.

Preferred Qualifications :

• Experience with tools like Apache Airflow, Azure Data Factory, or similar.
• Familiarity with CI/CD and DevOps in the context of data engineering.
• Knowledge of data governance, cataloging, and access control principles.

Skills : Python,Sql,Aws,Azure, Hadoop