Lead Data Engineer

GSPANN Technologies, Inc; Canada; On-site; CAD 80,000 - 110,000; Full time

Benefits offered by this job

Competitive benefits

Educational assistance

Career growth opportunities

Job summary

An established industry player is seeking a talented data engineer with extensive experience in Python and Spark. This role involves developing and optimizing data processing pipelines, ensuring data quality, and working with various data stores. The company values its employees and offers competitive benefits and opportunities for career growth. Join a diverse team and work with some of the best brands globally, while making a significant impact in the field of data engineering. If you're passionate about data and eager to grow your career, this opportunity is perfect for you.

Qualifications

• 5+ years of experience in heavy volume data processing and data platforms.
• Strong proficiency in Python and Spark is a must-have.

Responsibilities

• Develop and maintain ETL workflows using Spark, Scala, and Python.
• Ensure data quality and accuracy in processing pipelines.

Skills

Python

Spark

ETL Workflows

Data Processing Pipelines

SQL

Scala

Data Quality

Deep Learning Libraries

Architectural Patterns

Tools

PyTorch

TensorFlow

Keras

Redis

Delta Lake

Job description

Get AI-powered advice on this job and more exclusive features.

About GSPANN

Headquartered in Milpitas, California (U.S.A.), GSPANN provides consulting and IT services to global clients, ranging from mid-size to Fortune 500 companies. With our experience in retail, high-technology, and manufacturing, we help our clients to transform and deliver business value by optimizing their IT capabilities, practices, and operations. Counting on our ten offices, including four global delivery centers, and approximately 1400 employees globally, we offer the intimacy of a boutique consultancy with capabilities of a large IT services firm.

Location: Canada (Open to Support US PST Hours)

Job Type: Long Term

Interview Mode:

1. Coding Test will be required on SQL, Python/Pyspark/Scala.
2. Internal Video Call Interview.
3. Client Video Call.

Required Skills & Experience:

• Hands-on code mindset with deep understanding in technologies/skillset and an ability to understand larger picture.
• Sound knowledge to understand Architectural Patterns, best practices and Non-Functional Requirements.
• Overall, 5+ years of experience in heavy volume data processing, data platform, data lake, big data, data warehouse, or equivalent.
• 5+ years of experience with strong proficiency in Python and Spark (must-have).
• 3+ years of hands-on experience in ETL workflows using Spark, Scala and Python.
• 3+ years of experience with large-scale data loads, feature extraction, and data processing pipelines in different modes – near real time, batch, realtime.
• Solid understanding of data quality, data accuracy concepts and practices.
• 2+ years of experience working with Python deep learning libraries like PyTorch, Tensorflow, Keras.
• Experience integrating with various data stores, including: SQL/NoSQL databases, In-memory stores like Redis, Data lakes (e.g., Delta Lake).

Working at GSPANN

GSPANN is a diverse, prosperous, and rewarding place to work. We provide competitive benefits, educational assistance, and career growth opportunities to our employees. Every employee is valued for their talent and contribution. Working with us will give you an opportunity to work globally with some of the best brands in the industry.

The company does and will take affirmative action to employ and advance in the employment of individuals with disabilities and protected veterans and to treat qualified individuals without discrimination based on their physical or mental disability status. GSPANN is an equal opportunity employer for minorities/females/veterans/disabled.

Seniority level

Mid-Senior level

Employment type

Contract

Job function

Information Technology