Senior Data Engineer-AWS

InfoCepts; Pune District; On-site; INR 1,200,000 - 2,400,000; Full time

Job summary

InfoCepts is seeking an experienced data engineering professional with strong Big Data and cloud skills to join our Pune delivery team. The role focuses on technology assessment, design, and building data pipelines using AWS and Databricks, with emphasis on ETL workflows and complex transformations.

Ideal candidates will have 4+ years in IT, 3+ years in AWS projects, and solid Python/SQL expertise. Bachelor’s degree required; master’s a plus.

Qualifications

• 4+ years of IT experience.
• 3+ years of relevant AWS-related project experience.
• Bachelor’s degree in computer science, engineering, or related field; master's degree is a plus.

Responsibilities

• Technology assessment and design; study existing technology landscape and understand data integration frameworks.
• Documentation and stakeholder communication; project documentation adheres to quality guidelines & schedules.
• Process improvement and automation; suggest automation to improve existing processes and assist junior engineers.
• Training and knowledge sharing; create and deliver technology-focused training plans and sessions.
• Develop ETL workflows with complex transformations (SCD, deduplication, aggregations) and build data pipelines in AWS and Databricks.

Education

Bachelor’s degree in computer science, engineering, or related field

Master's degree (plus)

Tools

Airflow

Databricks

Spark

SQL

Python

AWS (S3, EC2, EMR, Athena, Glue, Lambda)

Snowflake

Redshift

Delta Lake

Hadoop/Hive

Job description

This position requires a sound knowledge on Big Data using Cloud technologies such as such as Databricks, EMR, Athena, PySpark, S3, AWS Lambda, Databricks, Snowflake etc. Strong foundation on database concepts and SQL is also required.

Key Result Areas and Activities:

• Technology Assessment and Design

Study existing technology landscape and understand current data integration frameworks and do impact assessment for the requirements.

• Documentation and Stakeholder Communication

Project documentation adheres to quality guidelines & schedules.

Works hand in hand with Architect & PM for successful delivery of project and provide estimation, scoping, scheduling assistance.

• Process Improvement and Automation

Suggests automation to improve existing processes.

Assist junior Data Engineers by providing expert advice or troubleshooting steps whenever required.

• Training and Knowledge Sharing

Create technology-focused training plans whenever required.

Deliver technology-focused training sessions with team members whenever required.

• In-depth knowledge of the following AWS services: S3, EC2, EMR, Athena, AWS Glue, Lambda
• Experience with at least one MPP database: AWS Redshift, Snowflake, SingleStore
• Proficiency in Big Data technologies: Apache Spark, Databricks
• Must have strong programming skills in Python
• Responsible for building data pipelines in AWS And Databricks
• Experience with Big Data table formats, such as Delta Lake (open source)
• Must have very strong SQL skills
• Experience with orchestration tools like Apache Airflow
• Expertise in developing ETL workflows with complex transformations such as SCD, deduplications, aggregations, etc.
• Should be a quick and self-learner, ready to adapt to new AWS services or Big Data technologies as required
• Strong Understanding with Media domain
• Strong understanding of data warehousing concepts
• Cloud Databases – Snowflake, AWS Aurora
• Big Data – Hadoop, Hive
• Associate Level or Professional Level AWS Certification or Databricks Certification

Qualifications:

• Overall, 4+ years of IT Exp
• 3+ years of relevant experience in AWS related project
• Bachelor’s degree in computer science, engineering, or related field (master's degree is a plus)
• Demonstrated continued learning through one or more technical certifications or related methods.

Qualities:

• Hold strong technical knowledge and experience
• Should have the capability to deep dive and research in various technical related fields
• Self-motivated and focused on delivering outcomes for a fast-growing team and firm
• Able to communicate persuasively through speaking, writing, and client presentations
• Able to consult, write, and present persuasively
• Able to work in a self-organized and cross-functional team
• Able to iterate based on new information, peer reviews, and feedback
• Prior experience of working in a large media company would be added advantage