Senior Technical Specialist - AWS/AWS Lambda/AWS Glue/AWS SNS

Datavail Career Site; Bengaluru; On-site; INR 3,500,000 - 6,000,000; Full time

Job summary

Datavail is seeking a Senior Data Engineer / AWS Data Engineer with 10-12 years of hands-on experience to design, develop, and maintain scalable data pipelines and cloud-native data solutions on AWS. You will integrate cloud and on-premises sources, optimize data processing, and support enterprise-scale platforms.

The role emphasizes strong Python/PySpark abilities, SQL optimization, and experience with Redshift, RDS, and ETL tools.

Qualifications

• Bachelor's or Master's degree in CS/IT/Engineering or related field.
• 10-12 years of professional experience in data engineering or related tech.
• Strong problem-solving and analytical skills.
• Excellent communication and collaboration skills.
• Ability to work independently and in cross-functional teams.

Responsibilities

• Design, develop, and maintain scalable data pipelines and ETL/ELT workflows.
• Develop cloud-native data solutions using AWS services (Lambda, Glue, SNS) and orchestration.
• Build and optimize data processing jobs with Python and PySpark.
• Develop complex SQL queries, stored procedures, and data transformations.
• Work with Amazon Redshift and Amazon RDS for storage and analytics.
• Develop and maintain SSIS packages and workflows.
• Use SSMS for database development and optimization.
• Implement infrastructure/configuration with Terraform and YAML.
• Use Git for source control and branching.
• Containerize workloads with Podman and develop in VS Code.
• Implement event-driven workflows using AWS services.
• Monitor, troubleshoot, and optimize data pipelines.
• Ensure data quality, reliability, security, and performance across platforms.
• Collaborate with architects, developers, analysts, and stakeholders.

Skills

Data engineering

ETL/ELT

Problem-solving

Analytical skills

Communication

Collaboration

Education

Bachelor's or Master's in CS/IT/Engineering

Tools

AWS

Python

PySpark

SQL

Redshift

RDS

SSIS

SSMS

Terraform

YAML

Git

Podman

VS Code

Job description

Job Title: Senior Technical Specialist

Education: Any Graduate

Experience: 10+years

Location: Bengaluru, Mumbai, Hyderabad.

We are looking for an experienced Senior Data Engineer / AWS Data Engineer with 10-12 years of hands-on experience in designing, developing, and maintaining scalable data engineering solutions. The ideal candidate should have strong expertise in AWS data services, Python, PySpark, SQL, ETL, infrastructure as code, and data orchestration.The candidate will be responsible for building robust data pipelines, integrating cloud and on-premises data sources, optimizing data processing workflows, and supporting enterprise-scale data platforms.

Technology Required Expertise

• AWSLambda, Glue, SNS, orchestration and related data services

• PythonStrong hands-on experience

• PySparkStrong hands-on experience

• SQLAdvanced SQL, query optimization, data transformation

• RedshiftDevelopment, optimization, data loading

• RDSDatabase development and management

• SSISETL development and maintenance

• SSMSSQL Server development and troubleshooting

• TerraformInfrastructure as Code

• YAMLConfiguration/automation

• GitVersion control and source code management

• PodmanContainerization

• VS CodeDevelopment environment

Good to Have

• Experience with Connect

• Experience with TOLSE / TOLENT

• Experience working in CI/CD and DevOps practices.

• Experience working in Agile/Scrum environments.

• Knowledge of cloud security, IAM, monitoring, and logging.

Qualifications

• Bachelor's or Master's degree in Computer Science, Information Technology, Engineering, or a related field.

• 10-12 years of professional experience in data engineering, ETL, cloud data platforms, or related technologies.

• Strong problem-solving and analytical skills.

• Excellent communication and collaboration skills.

• Ability to work independently as well as within a cross-functional team.

Target Profile: Senior Data Engineer / AWS Data Engineer / Lead Data Engineer with 10-12 years of experience

Key Responsibilities

• Design, develop, and maintain scalable data pipelines and ETL/ELT workflows.
• Develop cloud-native data solutions using AWS services, including Lambda, Glue, SNS, and orchestration services.
• Build and optimize data processing jobs using Python and PySpark.
• Develop complex SQL queries, stored procedures, and data transformations.
• Work with Amazon Redshift and Amazon RDS for data storage, processing, and analytics.
• Develop and maintain ETL packages and workflows using SSIS.
• Use SSMS for database development, troubleshooting, and performance optimization.
• Implement infrastructure and configuration using Terraform and YAML.
• Work with Git for source code management, branching, merging, and version control.
• Containerize and manage development workloads using Podman.
• Use VS Code as the primary development environment.
• Implement event-driven and automated workflows using AWS services such as Lambda, SNS, and orchestration frameworks.
• Monitor, troubleshoot, and optimize data pipelines and AWS workloads.
• Ensure data quality, reliability, security, and performance across data platforms.
• Collaborate with architects, developers, business analysts, and other stakeholders to deliver data solutions.
• Participate in code reviews, technical discussions, and production support activities.