Lead Data Engineer with PySpark

Infinity Outsourcing; Mount Laurel Township (NJ); On-site; USD 67,502 - 77,145; Full time

Job summary

A leading data engineering firm is seeking a Lead Data Engineer with expertise in PySpark to manage data platforms and architectures. This role demands 12+ years of experience, a strong knowledge of ETL and Data Warehousing, and the ability to lead multiple projects. The candidate should possess hands-on skills in using Pyspark and must be adept at working with both structured and unstructured data. Strong organizational and communication skills are essential for success in this dynamic environment.

Qualifications

• 12+ years of experience in data engineering and architecture.
• Hands-on experience in PySpark is mandatory.
• Must possess strong communication and collaboration skills.

Responsibilities

• Lead the development of data platforms and architectures.
• Coordinate with various technical teams for project management.
• Own the end-to-end development lifecycle including testing and deployment.

Skills

PySpark

Databricks

Data Warehousing

ETL

SQL Server

Data Integration

Cloud Architecture

GitHub

Job description

Mount Laurel Township, NJ, United States

About the job Lead Data Engineer with PySpark

Requisition Title :Lead Data Engineer with PySpark ( 1 Day Onsite)

Locations Mt Laurel, NJ & Charlotte, NC

Client : TD Bank

Duration: Full Time /C2H

Salary pay rate to candidates : $49-$56/hr W2

Experience : 12+ Years

Primary Tech skills =Pyspark, Databricks (Hands-on experience mandatory)

MUST HAVE SKILLS :

Techno-Managerial covers the entire life cycle of data warehouse and data cloud environment with experience in building data platforms development.

Technical co-ordination with Product Group Technical Lead, Architecture team, Data engineering team, BSA, and QE Team for the assigned projects.

Ability to drive more than one project in parallel.

Experience in building end to end architecture for Data Lakes, Data Warehouses and Data Marts

Experience of DWH, Data Integration, Cloud, Architecture, Design, Data Modelling

Hands-on experience in Pyspark is a must.

Experience working with structured and unstructured data.

Extensive knowledge of ETL and Data Warehousing concepts, strategies, methodologies

Experience in data ingestion, preparation, integration, and operationalization techniques in optimally addressing the data requirements.

Experience in relational data processing technology like MS SQL, Delta Lake, Spark SQL, SQL Server

Experience with Orchestration tools and GitHub

Experience to own end-to-end development, including coding, testing, debugging and deployment.

Must be team oriented with strong communication, collaboration, prioritization, and adaptability skill

Need to be an individual performer as well as a Lead.