Data Engineer

People's Growth HR Solutions; Bengaluru; On-site; INR 1,000,000 - 1,500,000; Full time

Job summary

A recruitment firm is seeking a Data Engineer to join their Data Science team in Bengaluru, India. The role involves designing and maintaining ETL processes for data analytics and machine learning. Candidates should have a Bachelor's degree in Computer Science and over 4 years of experience in software development, particularly in ETL techniques and cloud platforms. Strong programming skills in Python, Java, or Scala are essential, along with excellent analytical and communication skills.

Qualifications

• 4+ years of experience in software development, especially in product startup environments.
• Experience as a Data Engineer specializing in ETL processes.
• Proficiency in transforming data from public web sources.

Responsibilities

• Design and maintain ETL processes for various data sets.
• Collaborate with teams to understand data requirements.
• Implement data quality checks and governance standards.

Skills

ETL techniques

Python

Data modeling

SQL

Data governance

Cloud platforms (AWS, Azure, GCP)

Big data technologies (Hadoop, Spark, Kafka)

Data scraping

Analytical skills

Communication skills

Education

Bachelor's degree in Computer Science or related field

Job description

Timings: Full Time (As per company timings)

Notice Period: within 15 days or immediate joiner

Experience: 4+ Years

Interview round: 1st round + Assignment round + Final F2F round.

Basic Qualifications

• Bachelor's degree in Computer Science, Information Technology, or a related field.
• Prior professional, non-internship experience in software development, specifically within product startup environments.
• Proven experience as a Data Engineer with expertise in ETL techniques is a MUST
• Strong programming skills in languages such as Python, Java, or Scala.
• Skillset to scrape and transform data off of the publicly available web sources
• Experience with cloud-based data platforms (e.g., AWS, Azure, GCP).
• Proficiency in SQL and experience working with relational and non-relational databases.
• Knowledge of data warehousing concepts and architectures.
• Familiarity with big data technologies such as Hadoop, Spark, and Kafka.
• Experience with data modeling tools and techniques.
• Excellent problem-solving and analytical skills.
• Strong communication and collaboration skills.

Preferred Qualifications

• Master's degree or equivalent in Computer Science/ Data Science
• Knowledge of data streaming and real-time processing.
• Familiarity with data governance and security best practices.

About the role

As a Data Engineer for our Data Science team, you will play a critical role in helping the team to enrich and maintain the central repository of datasets which is leveraged for carrying out advanced data analytics and machine learning techniques to extract actionable insights from financial and market data. You will work closely with cross-functional teams to develop and implement robust data pipelines that will automate the updation of data in our cloud-based repository in a ready to use form thereby increasing data accessibility for the entire organization.

Key Responsibilities

ETL Development

• Design, develop, and maintain efficient ETL processes for multi-scale data sets.
• Implement and optimize data transformation and validation processes to ensure data accuracy and consistency.
• Collaborate with cross-functional teams to understand data requirements and business logic.
• Architect, build and maintain scalable and high-performance data pipelines
• Evaluate and implement new technologies to enhance data pipeline efficiency and reliability
• Pipelines for extracting data through scraping for ad-hoc sector specific datasets
• Develop and implement data models to support analytics and reporting needs.
• Optimize database structures for performance and scalability.

Data Quality and Governance

• Implement data quality checks and governance processes to ensure data integrity.
• Collaborate with stakeholders to define and enforce data quality standards.

Documentation and Communication

• Document ETL processes, data models, and other relevant information.
• Communicate complex technical concepts to non-technical stakeholders effectively.

Cross-functional collaboration

• Collaborate internally with the Quant team and developers to lay and optimize the data pipelines and externally with the stakeholders to understand the business requirements for the enrichment of the cloud database