Senior Big Data Engineer - Spark, Hadoop, Python

TechWish; McLean (VA); On-site; USD 90,000 - 130,000; Full time

Job summary

TechWish is seeking a seasoned data engineer to design, build, and maintain large-scale data processing pipelines using Hadoop, Spark, Python, and Scala. You will implement ingestion, storage, transformation, and analysis of scalable solutions while ensuring reliability and data quality across production systems.

The role requires collaboration with data scientists and analysts to enable data-driven decision making, ongoing learning, and staying current with industry trends in big data

Qualifications

• Bachelor's degree in Computer Science, Information Systems or related discipline with at least five (5) years of related experience
• Demonstrated technical expertise in object-oriented and database technologies
• Experience with developing enterprise-grade solutions in an Agile environment
• Strong knowledge of test automation, build automation and configuration management
• Excellent written and verbal technical communication skills
• Ability to work in a fast-paced environment
• Experience with Java, Scala or Python
• Experience with Big Data tools like Hadoop, Spark, Hive & Trino

Responsibilities

• Design, develop, and maintain large-scale data processing pipelines using Big Data technologies
• Implement data ingestion, storage, transformation, and analysis of scalable, efficient solutions
• Stay current with industry trends to improve data architecture
• Collaborate with cross-functional teams to translate business requirements into technical solutions
• Optimize and enhance pipelines for performance, scalability, reliability
• Develop automated testing frameworks and continuous testing for data quality
• Conduct unit, integration, and system testing for robustness
• Work with data scientists and analysts to support data-driven decisions
• Monitor and troubleshoot data pipelines in production environments

Skills

Big Data

Spark

Python

Scala

SQL

Cloud AWS

Data Pipelines

Testing & QA

Education

Bachelor's degree in CS/IS or related

5+ years of related experience

Tools

Hadoop

Hive

Trino

CI/CD

Job description

TechWish is seeking a seasoned data engineer to design, build, and maintain large-scale data processing pipelines using Hadoop, Spark, Python, and Scala. You will implement ingestion, storage, transformation, and analysis of scalable solutions while ensuring reliability and data quality across production systems.

The role requires collaboration with data scientists and analysts to enable data-driven decision making, ongoing learning, and staying current with industry trends in big data