Data Engineer

Acldigital; Pune District; On-site; INR 1,000,000 - 1,500,000; Full time

Job summary

A leading tech company in Pune is seeking an experienced AWS Data Engineer to design and optimize data infrastructure. Ideal candidates should have over 3 years of experience with AWS tools like Glue ETL and a solid background in data pipelines and warehousing. This position involves working with various stakeholders to address data needs and improve data processing capabilities.

Qualifications

• 3+ years of experience in a Data Engineer role in a cloud-native ecosystem.
• Experience in implementing data pipelines for both streaming and batch integrations.
• Experience in both Relational and NoSQL databases.

Responsibilities

• Design, Build and Operate data extraction and loading infrastructure.
• Optimize RDBMS engines in the cloud.
• Collaborate with internal stakeholders on data-related technical issues.

Skills

Data pipelines (ELT/ETL)

Data warehousing

Dimensional modeling

Problem-solving

AWS Glue ETL

Python

Advanced SQL

Education

Bachelor's degree in Computer Science, Mathematics, or related field

Tools

AWS Glue

Redshift

Lambda

Spark

DynamoDB

Job description

We are seeking an AWS-Data Engineer with strong experience across the entire Cloud Data stack. The ideal candidate should have extensive experience in data pipelines (ELT/ETL), data warehousing and dimensional modeling, and curation of data sets for Data Scientists and Business Intelligence users. Experience in building Data Lake. This candidate should also have excellent problem-solving ability dealing with large volumes of data and various data sources.

Key Responsibilities:

• Design, Build and Operate the infrastructure required for optimal extraction, transformation, and loading of data from a wide variety of data sources using SQL, cloud migration tools, and ‘big data’ technologies.
• Optimize various RDBMS engines in the cloud and solve customers' security, performance, and operation problems.
• Design, Build, and Operate large, complex data lakes that meet functional / non-functional business requirements.
• Optimize various data types of ingestion, storage, processing, and retrieval from near real-time events, and IoT, to unstructured data as images, audio, video, and documents, and in between.
• Work with customers' and internal stakeholders including the Executive, Product, Data, Software Development, and Design teams to assist with data-related technical issues and support their data infrastructure and business needs.
• Understand and Develop AI/ML models using AWS AI/ML services (e.g., Sage Maker, Rekognition , Comprehend) and collaborate with data scientists and engineers to implement machine learning solutions

Qualifications:

• 3+ years of experience in a Data Engineer role in a cloud-native eco-system.
• Bachelor (Graduate preferred) degree in Computer Science, Mathematics, Informatics, Information Systems, or another quantitative field.
• 3+ years working experience in AWS Glue ETL, Redshift, Step Function, Athena
• Experience in implementing data pipelines for both streaming and batch integrations using tools/frameworks like Glue ETL, Lambda, Spark, Spark Experience in Lake Formation Orchestration, Maintain and enhance models for SDLC using CI/CD pipelines in AWS.
• Experience in Relational and NoSQL databases, such as MySQL or Postgres and DynamoDB.
• Functional and scripting languages: Python, Advanced SQL.
• Experience building and optimizing ‘big data’ pipelines, architectures, and data sets.