PySpark Consultant Contract C2C jobs Urgent Need

Tech Mirrors; Irving (TX); On-site; USD 96,000 - 152,000; Full time

Job summary

Tech Mirrors in Irving, TX is seeking a PySpark Consultant to design and implement large-scale data pipelines using PySpark, Hadoop, Hive and Kafka. You will translate source PySpark code, ensure data quality and performance, and collaborate with data ingestion and DevOps teams.

The ideal candidate will have 4+ years of big data development experience, strong Python and SQL skills, and proven ability to lead data projects while maintaining data security and governance standards.

Qualifications

• 4+ years in big data development with Spark/Hadoop.
• Proficient in Python, PySpark and SQL.
• Experience with data quality and governance.
• Certification in big data or cloud tech preferred.

Responsibilities

• Design and implement PySpark data pipelines.
• Coordinate with ingestion and DevOps teams.
• Ensure data quality and security compliance.
• Troubleshoot PySpark apps and workflows.

Skills

Python

PySpark

SQL

Data Pipelines

DevOps

Communication

Leadership

Analytics

Troubleshooting

Tools

Hadoop

Hive

Kafka

Job description

PySpark Consultant
Contract
Irving TX

Job Description

• Experience with big data processing and distributed computing systems like Spark.
• Implement ETL pipelines and data transformation processes.
• Ensure data quality and integrity in all data processing workflows.
• Troubleshoot and resolve issues related to PySpark applications and workflows.
• Understand source, dependencies and data flow from converted PySpark code.
• Strong programming skills in Python and SQL.
• Experience with big data technologies like Hadoop, Hive, and Kafka.
• Understanding of data warehousing concepts and relational databases like SQL.
• Demonstrate and document code lineage.
• Integrate PySpark code with frameworks such as Ingestion Framework, DataLens, etc.,
• Ensure compliance with data security, privacy regulations, and organizational standards.
• Knowledge of CI/CD pipelines and DevOps practices.
• Strong problem-solving and analytical skills.
• Excellent communication and leadership abilities.

Qualifications

• 4+ years of experience in big data development, Hadoop , Hive & Spark framework.
• Good to have experience in SAS.
• Strong Python, PySpark Development and SQL knowledge.
• Certification in big data or cloud technologies is preferred.