Technology Lead - PySpark & Big Data

Infosys; Mississauga; On-site; CAD 93,000 - 123,000; Full time

Benefits offered by this job

Databricks experience

Job summary

Infosys is seeking an experienced PySpark Developer to design, develop, and optimize scalable bigdata solutions in Mississauga, Ontario. You will build high-performance batch and real-time data pipelines using Spark, Kafka, and Hadoop ecosystem, collaborating with data engineers, architects, and stakeholders.

The role requires 4+ years in IT and Big Data, with strong PySpark, Spark, Scala, and SQL skills. Relocation within commuting distance to Mississauga or willingness to relocate is expected.

Qualifications

• Bachelor’s degree or foreign equivalent from an accredited institution.
• At least 4 years of IT experience.
• At least 4 years of Big Data technologies experience.
• Strong expertise in: Apache Spark (Core, SQL, DataFrames, RDDs), Scala, PySpark.
• Hands-on with Kafka, Hadoop ecosystem (HDFS, Hive, Impala), NoSQL databases (HBase, MongoDB, Couchbase).
• Proficiency in SQL and data modeling.

Responsibilities

• Design and develop large-scale data processing pipelines using Apache Spark (Scala & PySpark).
• Build and optimize batch and real-time data processing workflows using Spark, Kafka, and Hadoop ecosystem.
• Develop Spark applications using RDDs, DataFrames, and Spark SQL for complex transformations.
• Develop and optimize PySpark applications leveraging joins, Spark DAG execution flow, stage optimization, transformation techniques, and streaming with dynamic allocation and failover handling.
• Implement streaming pipelines using Kafka and Spark Streaming / Structured Streaming.
• Develop and maintain HDFS, Hive, NoSQL and Impala-based data lake solutions.
• Convert existing SQL/Hive workloads into optimized Spark jobs for improved performance.
• Work with ETL pipelines to ingest, cleanse, transform, and process large datasets.
• Optimize performance through partitioning, caching, serialization, and tuning techniques.
• Handle data formats such as Parquet, ORC, Avro, JSON.
• Integrate multiple data sources including streaming systems, flat files RDBMS, and APIs.
• Collaborate with cross-functional teams to translate business requirements into scalable technical solutions.
• Ensure data quality, reliability, and performance monitoring across pipelines.
• Participate in code reviews, design discussions, and best practices implementation.

Skills

Apache Spark

Spark SQL

DataFrames

PySpark

Scala

Kafka

Hadoop

NoSQL

SQL & Data Modeling

Education

Bachelor's degree or foreign equivalent

Tools

Databricks

AWS

GCP

Job description

Infosys is seeking an experienced PySpark Developer to design, develop, and optimize scalable bigdata solutions in Mississauga, Ontario. You will build high-performance batch and real-time data pipelines using Spark, Kafka, and Hadoop ecosystem, collaborating with data engineers, architects, and stakeholders.

The role requires 4+ years in IT and Big Data, with strong PySpark, Spark, Scala, and SQL skills. Relocation within commuting distance to Mississauga or willingness to relocate is expected.