Senior PySpark Data Engineer

VDart Inc; Irving (TX); On-site; USD 120,000 - 160,000; Full time

Job summary

VDart Inc. in Irving, TX is seeking an experienced Big Data Developer to design and implement PySpark-based data processing pipelines. You will ensure data quality, troubleshoot PySpark workflows, and integrate with ingestion and data-lens frameworks while maintaining data security standards.

You should have 4+ years in big data with Hadoop, Hive and Spark, strong Python/SQL skills, and a proven ability to lead, communicate, and document data lineage across complex data flows.

Qualifications

• 4+ years of experience in big data development, Hadoop, Hive & Spark.
• Strong Python and SQL knowledge.
• Experience with PySpark and Spark frameworks.
• Knowledge of data security and CI/CD practices.

Responsibilities

• Design and implement ETL pipelines and data transformations.
• Troubleshoot PySpark applications and workflows.
• Ensure data quality and integrity across processing workflows.
• Document data lineage and data flow.
• Understand sources and dependencies of converted PySpark code.
• Integrate PySpark code with Ingestion Framework and DataLens where applicable.

Skills

Big data development

Python

SQL

PySpark

Spark framework

Hadoop

Hive

Kafka

Data lineage

CI/CD / DevOps

Leadership

Communication

Tools

PySpark

Spark

Hadoop

Hive

Kafka

Job description

VDart Inc. in Irving, TX is seeking an experienced Big Data Developer to design and implement PySpark-based data processing pipelines. You will ensure data quality, troubleshoot PySpark workflows, and integrate with ingestion and data-lens frameworks while maintaining data security standards.

You should have 4+ years in big data with Hadoop, Hive and Spark, strong Python/SQL skills, and a proven ability to lead, communicate, and document data lineage across complex data flows.