Senior Data Engineer/Architect

Infosys; Bengaluru; On-site; INR 1,200,000 - 1,500,000; Full time

Benefits offered by this job

Challenging projects

Career growth

Mentorship

Global opportunities

Competitive compensation

Job summary

A global technology consulting company in Bengaluru is seeking a Software Developer focused on AWS technologies and Scala programming. This mid-senior level position requires over 5 years of experience in developing Spark Jobs and building data pipelines. Competitive compensation and opportunities for challenging projects are provided.

Qualifications

• 5+ years of hands-on experience in developing, testing, and debugging Spark Jobs using Scala.
• In-depth knowledge of Spark Core and working with RDDs.
• Experience in Designing and building data pipelines.

Responsibilities

• Create, maintain, audit, and improve systems.
• Test both hardware and software systems.
• Write diagnostic programs and design software for efficiency.

Skills

AWS Glue

AWS Redshift/Spectrum

S3

API Gateway

Spark SQL

Python

Scala

Education

BE/B.Tech in Computer Science or equivalent

Tools

Hadoop

Airflow

Git

Job description

Overview

As a Software Developer you will work in a constantly evolving environment, due to technological advances and the strategic direction of the organization you work for. You will create, maintain, audit, and improve systems to meet particular needs, often as advised by a systems analyst or architect, testing both hard and software systems to diagnose and resolve system faults. The role also covers writing diagnostic programs and designing and writing code for operating systems and software to ensure efficiency. When required, you will make recommendations for future developments.

Benefits

• Challenging Projects: Work on cutting-edge projects and solve complex technical problems.
• Career Growth: Advance your career quickly and take on leadership roles.
• Mentorship: Learn from experienced mentors and industry experts.
• Global Opportunities: Work with clients from around the world and gain international experience.
• Competitive Compensation: Receive attractive compensation packages and benefits.
• If you\'re passionate about technology and want to work on challenging projects with a talented team, becoming an Infosys Power Programmer could be a great career choice.

Mandatory Skills

• AWS Glue, AWS Redshift/Spectrum, S3, API Gateway, Athena, Step and Lambda functions
• Experience in Extract Transform Load (ETL) and Extract Load & Transform (ELT) data integration pattern.
• Experience in designing and building data pipelines.
• Development experience in one or more object-oriented programming languages, preferably Python

Job Specs

• 5+ years of in-depth hands-on experience of developing, testing, deployment and debugging of Spark Jobs using Scala in Hadoop Platform
• In-depth knowledge of Spark Core, working with RDDs, Spark SQL
• In-depth knowledge on Spark Optimization Techniques and Best practices
• Good Knowledge of Scala Functional Programming: Try, Option, Future, Collections
• Good Knowledge of Scala OOPS: Classes, Traits and Objects (Singleton and Companion), Case Classes
• Good Understanding of Scala Language Features: Type System, Implicit/Givens
• Hands on experience of working in Hadoop Environment (HDFS/Hive), AWS S3, EMR
• Working experience on Workflow Orchestration tools like Airflow, Oozie
• Working with API calls in Scala
• Understanding and exposure to file formats such as Apache AVRO, Parquet, JSON
• Good to have knowledge of Protocol Buffers and Geospatial data analytics.
• Writing Test cases using frameworks such as scalatest.
• Good Knowledge of Build Tools such as: Gradle & SBT in depth
• Experience on using GIT, resolving conflicts, working with branches.
• Good to have worked on some workflow systems as Airflow
• Strong programming skills using data structures and algorithms.
• Excellent analytical skills

Qualifications

• 7-10 Yrs in the industry
• BE/B.tech CS or equivalent

Seniority level

• Mid-Senior level

Employment type

• Full-time

Job function

• IT Services and IT Consulting