Big Data Engineer

Qcentrio; Pune District; On-site; INR 2,500,000 - 3,600,000; Full time

Job summary

Qcentrio is seeking a seasoned Data Engineer to drive data integration, modeling, and analytics across large-scale data platforms. You will design pipelines, implement data lake solutions on AWS, and collaborate with cross-functional teams including business and technology stakeholders.

The ideal candidate has 6+ years of experience, strong Python and SQL skills, and hands-on experience with Spark/PySpark, Hive, and Airflow. Immediate notice is preferred, with Pan India work footprint.

Qualifications

• 6+ years of relevant experience in data engineering or analytics.
• Experience building data lake solutions on AWS (S3, EMR), Hive, PySpark.
• Proficient in SQL and Python scripting.
• Experience with GitHub and modern DevOps practices.
• Experience with workflow scheduling tools such as Airflow.
• Strong knowledge of AWS cloud services and data platforms.

Responsibilities

• Participate in all phases of the software development lifecycle from requirements to deployment.
• Craft scalable, efficient data solutions and ensure data quality at scale.
• Analyze source and target data, map transformations, and implement pipelines.
• Collaborate with clients and onsite coordinators to deliver data projects.
• Design and implement data features with stakeholders across teams.
• Troubleshoot data issues and perform root-cause analysis to prevent recurrence.

Skills

Big Data

Python

SQL

Spark/Pyspark

AWS Cloud

Education

Bachelor's degree in computer science

Tools

GitHub

Airflow

Databricks

Hive

S3

EMR

Job description

Work Location : Pan India

Experience : 6+ Years

Notice Period : Immediate - 30 days

Mandatory Skills : Big Data, Python, SQL, Spark/Pyspark, AWS Cloud

JD and required Skills & Responsibilities :

• Actively participate in all phases of the software development lifecycle, including requirements gathering, functional and technical design, development, testing, roll-out, and support.
• Solve complex business problems by utilizing a disciplined development methodology.
• Produce scalable, flexible, efficient, and supportable solutions using appropriate technologies.
• Analyse the source and target system data. Map the transformation that meets the requirements.
• Interact with the client and onsite coordinators during different phases of a project.
• Design and implement product features in collaboration with business and Technology stakeholders.
• Anticipate, identify, and solve issues concerning data management to improve data quality.
• Clean, prepare, and optimize data at scale for ingestion and consumption.
• Support the implementation of new data management projects and re-structure the current data architecture.
• Implement automated workflows and routines using workflow scheduling tools.
• Understand and use continuous integration, test-driven development, and production deployment frameworks.
• Participate in design, code, test plans, and dataset implementation performed by other data engineers in support of maintaining data engineering standards.
• Analyze and profile data for the purpose of designing scalable solutions.
• Troubleshoot straightforward data issues and perform root cause analysis to proactively resolve product issues.

Required Skills :

• 5+ years of relevant experience developing Data and analytic solutions.
• Experience building data lake solutions leveraging one or more of the following AWS, EMR, S3, Hive & PySpark
• Experience with relational SQL.
• Experience with scripting languages such as Python.
• Experience with source control tools such as GitHub and related dev process.
• Experience with workflow scheduling tools such as Airflow.
• In-depth knowledge of AWS Cloud (S3, EMR, Databricks)
• Has a passion for data solutions.
• Has a strong problem-solving and analytical mindset
• Working experience in the design, Development, and test of data pipelines.
• Experience working with Agile Teams.
• Able to influence and communicate effectively, both verbally and in writing, with team members and business stakeholders
• Able to quickly pick up new programming languages, technologies, and frameworks.
• Bachelor's degree in computer science