PySpark Consultant

VDart Inc; Irving (TX); On-site; USD 120,000 - 160,000; Full time

Job summary

VDart Inc. in Irving, TX is seeking an experienced Big Data Developer to design and implement PySpark-based data processing pipelines. You will ensure data quality, troubleshoot PySpark workflows, and integrate with ingestion and data-lens frameworks while maintaining data security standards.

You should have 4+ years in big data with Hadoop, Hive and Spark, strong Python/SQL skills, and a proven ability to lead, communicate, and document data lineage across complex data flows.

Qualifications

• 4+ years of experience in big data development, Hadoop, Hive & Spark.
• Strong Python and SQL knowledge.
• Experience with PySpark and Spark frameworks.
• Knowledge of data security and CI/CD practices.

Responsibilities

• Design and implement ETL pipelines and data transformations.
• Troubleshoot PySpark applications and workflows.
• Ensure data quality and integrity across processing workflows.
• Document data lineage and data flow.
• Understand sources and dependencies of converted PySpark code.
• Integrate PySpark code with Ingestion Framework and DataLens where applicable.

Skills

Big data development

Python

SQL

PySpark

Spark framework

Hadoop

Hive

Kafka

Data lineage

CI/CD / DevOps

Leadership

Communication

Tools

PySpark

Spark

Hadoop

Hive

Kafka

Job description

Experience with big data processing and distributed computing systems like Spark.

Implement ETL pipelines and data transformation processes.

Ensure data quality and integrity in all data processing workflows.

Troubleshoot and resolve issues related to PySpark applications and workflows.

Understand source, dependencies and data flow from converted PySpark code.

Strong programming skills in Python and SQL.

Experience with big data technologies like Hadoop, Hive, and Kafka.

Understanding of data warehousing concepts and relational databases like SQL.

Demonstrate and document code lineage.

Integrate PySpark code with frameworks such as Ingestion Framework, DataLens, etc.,

Ensure compliance with data security, privacy regulations, and organizational standards.

Knowledge of CI/CD pipelines and DevOps practices.

Strong problem-solving and analytical skills.

Excellent communication and leadership abilities. Qualifications:

4+ years of experience in big data development, Hadoop , Hive & Spark framework.

Good to have experience in SAS.

Strong Python, PySpark Development and SQL knowledge.

Certification in big data or cloud technologies is preferred.