Aws Data Engineer - Gen AI

Qentelli; Hyderabad; On-site; INR 1,500,000 - 2,100,000; Full time

Job summary

Qentelli is seeking an experienced AWS Data Engineer to design, build, and maintain scalable data pipelines on AWS, handling large volumes of structured and unstructured data for AI applications.

You will work with Python, SQL, AWS services (S3, Glue, Redshift, Athena, Lambda, EMR, Step Functions) and contribute to data warehousing, data lakes, and data quality. Collaboration with data scientists and DevOps is essential.

Qualifications

• 4+ years of data engineering experience.
• Hands-on with AWS data services and data pipelines.
• Experience with S3, Glue, Redshift, Athena, Lambda, IAM.
• ETL/ELT pipeline development experience.
• Strong knowledge of data warehousing and data lakes.
• Experience handling large datasets and batch/incremental processing.
• Monitoring, troubleshooting, and data quality enforcement.

Responsibilities

• Design, develop, and maintain scalable data pipelines on AWS.
• Build ETL/ELT processes to extract, transform, and load data.
• Develop data solutions using AWS services like S3, Glue, Redshift, Athena, Lambda, EMR.
• Write optimized SQL queries for processing and reporting.
• Create Python scripts for data processing and automation.
• Handle structured and unstructured data from various sources.
• Build data models and datasets for analytics and AI applications.
• Implement data validation, quality checks, and governance.
• Monitor pipelines, troubleshoot failures, and optimize performance and costs.
• Collaborate with data scientists and engineers and document pipelines.

Skills

AWS Data Engineer

Python

SQL

ETL/ELT

Data Warehousing

Generative AI

Data Governance

Tools

S3

Glue

Redshift

Athena

Lambda

IAM

EMR

Step Functions

Job description

JOB SUMMARY:

• We are looking for an experienced AWS Data Engineer with hands-on experience in building and maintaining data pipelines, data platforms, and cloud-based data solutions on AWS.
• The candidate should have strong experience in Python, SQL, AWS data services, ETL/ELT pipelines, and data warehousing, along with practical exposure to Generative AI applications.
• The role will involve working with large volumes of structured and unstructured data, building reliable data pipelines, preparing data for AI applications, and supporting solutions that use Generative AI.
• The ideal candidate should be comfortable working with business and technical teams, understanding data requirements, solving data quality issues, and delivering scalable data solutions.

ROLES AND RESPONSIBILITIES:

• Design, develop, and maintain scalable data pipelines on AWS.
• Build ETL/ELT processes to extract data from multiple sources, transform it, and load it into data platforms.
• Develop data solutions using AWS services such as S3, Glue, Lambda, Redshift, Athena, EMR, and Step Functions.
• Develop and optimize SQL queries for data processing and reporting.
• Write Python scripts and applications for data processing and automation.
• Work with structured and unstructured data from databases, APIs, files, and other sources.
• Build data models and prepare datasets for analytics and AI applications.
• Implement data validation and quality checks across data pipelines.
• Monitor data pipelines and troubleshoot failures, performance issues, and data-related problems.
• Improve pipeline performance, reliability, and cost efficiency.
• Work with large datasets and optimize data processing jobs.
• Integrate data from relational databases, cloud storage, APIs, and enterprise applications.
• Build and maintain data warehouses, data lakes, and related data processing solutions.
• Implement appropriate security, access controls, encryption, and data governance practices on AWS.
• Work with DevOps teams to deploy and manage data solutions across development, testing, and production environments.
• Create technical documentation for data pipelines, data models, and processes.
• Collaborate with data scientists, software engineers, analysts, and business teams to understand requirements.
• Support data preparation for Generative AI applications, including collecting, cleaning, transforming, and organizing relevant data.
• Work with text documents and other unstructured data used by AI applications.
• Build data pipelines that support AI-powered search, question-answering, and documentbased applications.
• Work with teams to store, process, and retrieve information required by Generative AI applications.
• Monitor the quality and accuracy of data used by AI applications.
• Participate in design discussions, code reviews, testing, and production support.

MANDATORY SKILLS:

• 4+ years of experience in Data Engineering.
• Strong hands-on experience with AWS data services.
• Strong experience with AWS S3, Glue, Redshift, Athena, Lambda, and IAM.
• Experience developing ETL/ELT data pipelines.
• Strong knowledge of data warehousing and data lake concepts.
• Experience working with large datasets.
• Good understanding of batch and incremental data processing.
• Experience with data pipeline monitoring and troubleshooting.

Programming & Database

• Strong programming experience in Python.
• Strong SQL skills, including joins, subqueries, CTEs, window functions, and query optimization.
• Experience with relational databases such as PostgreSQL, MySQL, Oracle, or SQL Server.
• Good understanding of data modeling and database design.
• Experience working with JSON, CSV, Parquet, and other common data formats.

Gen AI Experience

• Practical experience working on applications that use Generative AI.
• Experience preparing and processing documents or other unstructured data for AI applications.
• Understanding of how data is collected, cleaned, processed, stored, and retrieved for Generative AI use cases.
• Experience working with embeddings and vector-based search.
• Experience with at least one vector database or AWS-supported vector search solution.
• Experience integrating data pipelines with AI/Gen AI applications.
• Basic understanding of how document-based question-answering or search applications work.
• Experience working with APIs or cloud services used by Generative AI applications.

Engineering Practices

• Experience with Git and source-code management.
• Experience with CI/CD processes.
• Good understanding of AWS security and IAM.
• Experience with logging, monitoring, error handling, and production support.
• Strong problem-solving and debugging skills.

Preferred Skills

• Experience with AWS AI services.
• Experience with Amazon OpenSearch or other vector search technologies.
• Experience working with document processing pipelines.
• Experience with PDF, Word, HTML, XML, or text document processing.
• Experience with Spark or PySpark.
• Experience with AWS EMR.
• Experience with Apache Airflow or similar workflow orchestration tools.
• Experience with Terraform or CloudFormation.
• Experience with Docker and container-based deployments.
• Experience with REST APIs and API-based data integration