Senior Data Engineer with Pyspark

Hexplora

Rancho Cucamonga (CA)

Hybrid

USD 120,000 - 170,000

Full time

32 hours ago
Be an early applicant

Get a reply from this employer

Your resume and cover letter, tailored to exactly what they’re hiring for.

Get past ATS filters

Job summary

Infowave Systems is seeking a Senior Data Engineer (PySpark, ETL) to design, build, and maintain scalable data pipelines in the healthcare domain. The role blends development with operational oversight to ensure reliable performance.

You will work on migration and integration tasks across systems, leveraging PySpark, SSIS, and Cloudera Data Hub, while collaborating with data scientists and analysts to meet data requirements.

Qualifications

  • Strong experience in Data Engineering: designing, building, and maintaining ETL pipelines.
  • Proficiency with PySpark for large-scale processing.
  • Experience with SSIS and Cloudera Data Hub is a plus.

Responsibilities

  • Design, develop, and maintain scalable ETL pipelines using PySpark to process large datasets.
  • Collaborate with data scientists, analysts, and stakeholders to translate requirements into robust solutions.
  • Monitor, troubleshoot, and optimize data pipelines and infrastructure.
  • Manage data integration across healthcare systems, ensuring compliance with standards.
  • Leverage SSIS for ETL development across environments.
  • Integrate and transform data from multiple sources for quality and consistency.
  • Resolve data processing issues to minimize operational disruption.
  • Document best practices and workflows for pipeline efficiency.
  • Work with relational and non-relational databases to ensure smooth data flow and performance.

Skills

Data Engineering
PySpark
ETL pipelines
Relational databases
Data warehousing
Cloud platforms (AWS/GCP/Azure)
Healthcare data standards HL7/FHIR
Data pipeline monitoring

Tools

SSIS
Cloudera Data Hub

Job description

Location: Rancho Cucamonga,CA(Hybrid)

Job Title: Senior Data Engineer (PySpark, ETL,Cloudera Data Hub)

Job Description

We are looking for an experienced and motivated Data Engineer with expertise in PySpark to join our dynamic team. As a key member of our data engineering team, you will play a crucial role in designing, building, and maintaining scalable data pipelines that enable efficient data processing and analytics within the healthcare domain. This role will combine both development and administrative activities, making it essential that the candidate has experience not only in building robust data pipelines but also in overseeing their operational aspects to ensure performance, reliability, and optimization.

Key Responsibilities
  • Design, develop, and maintain scalable ETL pipelines using PySpark to process large datasets.
  • Collaborate with cross-functional teams (data scientists, analysts, business stakeholders) to understand data requirements and deliver high-quality solutions.
  • Work on administrative tasks, including monitoring, troubleshooting, and optimizing data pipelines and infrastructure.
  • Manage data integration across healthcare systems, ensuring compliance with relevant standards.
  • Leverage SSIS for ETL development and ensure smooth data movement across different environments.
  • Integrate and transform data from multiple sources, ensuring data quality and consistency.
  • Handle and resolve data processing issues, ensuring minimal disruption to operations.
  • Document best practices, processes, and workflows to maintain pipeline efficiency and scalability.
  • Work with both relational and non-relational databases, ensuring smooth data flow and optimized performance.
Required Skills & Qualifications
  • Strong experience in Data Engineering, with expertise in designing, building, and maintaining ETL pipelines.
  • Strong proficiency in PySpark for large-scale data processing and transformation.
  • Experience with ETL tools, particularly SSIS (SQL Server Integration Services).
  • Solid understanding of data modeling, relational databases, and data warehousing principles.
  • Cloudera Data Hub experience will be Advantage.
  • Experience working with cloud-based data storage and processing technologies (AWS, GCP, or Azure).
  • Familiarity with healthcare data standards, such as HL7 and FHIR, is highly desirable.
  • Proven ability in data pipeline monitoring, troubleshooting, and performance tuning.
  • Strong communication skills and the ability to work collaboratively with cross-functional teams.

Infowave Systems is an equal opportunity employer that is committed to diversity and inclusion in the workplace.

Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Senior PySpark Data Engineer - Healthcare ETL & Pipelines
Senior PySpark Data Engineer - Healthcare ETL & Pipelines

Hexplora • Rancho Cucamonga (CA)

Hybrid
USD 120,000 - 170,000
Data Engineer
Data Engineer

The Value Maximizer • United States

On-site
USD 90,000 - 120,000
PySpark Data Engineer
PySpark Data Engineer

Tata Consultancy Services Limited • Irving (TX)

On-site
USD 90,000 - 120,000
Discretionary Annual Incentive
Medical Coverage (Health, Dental & Vis
401K Plan
+1
Senior Data Engineer – Python, PySpark & AWS
Senior Data Engineer – Python, PySpark & AWS

SCI SOLUTIONS GLOBAL • Plano (TX)

On-site
USD 120,000 - 160,000
Data Engineer
Data Engineer

Tata Consultancy Services Limited • Irving (TX)

On-site
USD 125,000 - 140,000
Senior Spark ETL Engineer for Enterprise Data Solutions
Senior Spark ETL Engineer for Enterprise Data Solutions

Compunnel Inc. • Durham (NC)

On-site
USD 100,000 - 130,000
AWS Data Engineer
AWS Data Engineer

Tata Consultancy Services Limited • Irving (TX)

On-site
USD 90,000 - 110,000
Data Engineer ETL
Data Engineer ETL

Compunnel Inc. • Durham (NC)

On-site
USD 100,000 - 130,000
Senior Data Engineer: Scalable Pipelines & Data Quality
Senior Data Engineer: Scalable Pipelines & Data Quality

TechDigital Group • Burbank (CA)

On-site
USD 120,000 - 150,000
Data Engineer
Data Engineer

Ethereum Technologies LLC • Charlotte (NC)

On-site
USD 120,000 - 180,000