Big Data Engineer - Python

Qcentrio; Jaipur; On-site; INR 2,500,000 - 4,000,000; Full time

Job summary

Qcentrio in Jaipur is seeking an experienced Data Engineer with 7+ years to design and maintain scalable ETL pipelines and manage TB-scale datasets. You will build robust data infrastructure and collaborate closely with data science and software teams.

The role requires deep expertise in Python, SQL, Spark/Hadoop, and cloud platforms (Azure preferred). You will help implement data lake/lakehouse architectures and ensure data quality, security, and cost-efficient operations.

Qualifications

• 7–10 years of experience in bigdata engineering and scalable data infra.
• Advanced Python programming and strong SQL for complex data tasks.
• Hands-on with Spark, Hadoop, or similar distributed systems.
• Experience handling TB-scale datasets in production environments.
• Cloud development with Azure preferred, AWS or GCP also acceptable.
• Knowledge of data lake/data lakehouse architectures and ETL tuning.

Responsibilities

• Design, develop, and maintain scalable ETL/ELT pipelines for large data volumes.
• Model and structure data for performance, scalability, and usability.
• Work with cloud infra (Azure preferred) to optimize data workflows.
• Leverage Spark and Hadoop for large-scale data processing.
• Build data lake/lakehouse architectures aligned with best practices.
• Optimize ETL performance and manage cost-effective data ops.
• Collaborate with data science, analytics, and software engineering teams.
• Ensure data quality, integrity, and security across the data lifecycle.

Skills

Bigdata engineering

Python

SQL

Distributed computing

Cloud platforms (Azure/AWS/GCP)

Tools

Apache Spark

Hadoop

Apache Airflow

Azure Data Factory

Job description

Job description :

We are seeking an experienced and driven Data Engineer with 5+ years of hands-on experience in building scalable data infrastructure and systems. You will play a key role in designing and developing robust, high-performance ETL pipelines and managing large-scale datasets to support critical business functions. This role requires deep technical expertise, strong problem-solving skills, and the ability to thrive in a fast-paced, evolving environment.

Key Responsibilities :

• Design, develop, and maintain scalable and reliable ETL/ELT pipelines for processing large volumes of data (terabytes and beyond).
• Model and structure data for performance, scalability, and usability.
• Work with cloud infrastructure (preferably Azure) to build and optimize data workflows.
• Leverage distributed computing frameworks like Apache Spark and Hadoop for large-scale data processing.
• Build and manage data lake/lakehouse architectures in alignment with best practices.
• Optimize ETL performance and manage cost-effective data operations.
• Collaborate closely with cross-functional teams including data science, analytics, and software engineering.
• Ensure data quality, integrity, and security across all stages of the data lifecycle.

Required Skills & Qualifications :

• 7 to 10 years of relevant experience in bigdata engineering.
• Advanced proficiency in Python,
• Strong skills in SQL for complex data manipulation and analysis.
• Hands-on experience with Apache Spark, Hadoop, or similar distributed systems.
• Proven track record of handling large-scale datasets (TBs) in production environments.
• Cloud development experience with Azure (preferred), AWS, or GCP.
• Solid understanding of data lake and data lakehouse architectures.
• Expertise in ETL performance tuning and cost optimization techniques.
• Knowledge of data structures, algorithms, and modern software engineering practices.

Soft Skills :

• Strong communication skills with the ability to explain complex technical concepts clearly and concisely.
• Self-starter who learns quickly and takes ownership.
• High attention to detail with a strong sense of data quality and reliability.
• Comfortable working in an agile, fast-changing environment with incomplete requirements.

Preferred Qualifications :

• Experience with tools like Apache Airflow, Azure Data Factory, or similar.
• Familiarity with CI/CD and DevOps in the context of data engineering.
• Knowledge of data governance, cataloging, and access control principles.

Skills : Python,Sql,Aws,Azure, Hadoop