Big Data Engineer

Eliassen Group; Tysons (VA); Hybrid; Confidential; Full time

Benefits offered by this job

Medical insurance

Dental insurance

Vision insurance

401k with company matching

Life insurance

Job summary

Eliassen Group in Tysons Corner, VA is seeking a highly skilled Big Data Engineer to design, develop, and optimize large-scale data processing systems. You will work with cross-functional teams to architect data pipelines and ensure performance, scalability, and reliability of big data platforms in a hybrid onsite/remote setting.

Ideal candidates bring expertise in Hadoop, Spark, Hive, and Trino, with experience in Java, Scala, or Python.

Qualifications

• Bachelor's degree in Computer Science, Information Systems, or related discipline with at least five years of related experience, or equivalent training and work experience.
• Master's degree and Financial Services experience preferred.

Responsibilities

• Design, develop, and maintain large-scale data processing pipelines using Big Data technologies such as Hadoop, Spark, Python, and Scala.
• Implement data ingestion, storage, transformation, and analysis solutions that are scalable, efficient, and reliable.
• Stay current with industry trends and emerging Big Data technologies to improve the data architecture.
• Collaborate with cross-functional teams to translate business requirements into technical solutions.
• Optimize and enhance existing data pipelines for performance, scalability, and reliability.
• Develop automated testing frameworks and implement continuous testing for data quality assurance.
• Monitor and troubleshoot data pipelines in production to identify and resolve issues.
• Partner with data scientists and analysts to support data-driven decision-making.
• Write and maintain automated unit, integration, and end-to-end tests.

Skills

Java

Python

Scala

Big Data

Spark

Hadoop

Hive

Trino

AWS

SQL

AI tools

Education

Bachelor's degree in Computer Science

Master's degree (preferred)

Tools

GitHub Copilot

Q Developer

ChatGPT

Claude

Scrum

Kanban

Job description

Description

Hybrid 3 days onsite / 2 days remote in Tysons Corner, VA

Our client is seeking a highly skilled and experienced Big Data Engineer to design, develop, and optimize large-scale data processing systems. You will work with cross-functional teams to architect data pipelines, implement data integration solutions, and ensure the performance, scalability, and reliability of big data platforms. The ideal candidate will have expertise in distributed systems, cloud platforms, and modern big data technologies such as Hadoop and Spark.

We can facilitate w2 and corp-to-corp consultants. For our w2 consultants, we offer a great benefits package that includes Medical, Dental, and Vision benefits, 401k with company matching, and life insurance.

Rate: $65.00 to $70.00/hr. w2

Responsibilities

• Design, develop, and maintain large-scale data processing pipelines using Big Data technologies such as Hadoop, Spark, Python, and Scala.
• Implement data ingestion, storage, transformation, and analysis solutions that are scalable, efficient, and reliable.
• Stay current with industry trends and emerging Big Data technologies to improve the data architecture.
• Collaborate with cross-functional teams to translate business requirements into technical solutions.
• Optimize and enhance existing data pipelines for performance, scalability, and reliability.
• Develop automated testing frameworks and implement continuous testing for data quality assurance.
• Conduct unit, integration, and system testing to ensure the robustness and accuracy of data pipelines.
• Partner with data scientists and analysts to support data-driven decision-making.
• Write and maintain automated unit, integration, and end-to-end tests.
• Monitor and troubleshoot data pipelines in production to identify and resolve issues.
• Apply SQL skills including window functions, joins, aggregations, and handling of NULLs, duplicates, and ordering.
• Design and tune Apache Spark jobs, including partitioning, caching, broadcast joins, and troubleshooting DAG, stages, tasks, and executor performance.
• Leverage AWS services such as S3, EMR, Glue, Lambda, and Athena, including Spark with S3 file formats and consistency considerations.
• Write clean, modular, and performant Python or Scala code using functional programming concepts, and manage collections, concurrency, and memory.
• Employ AI tools and prompt engineering to enhance development workflows, analysis, and continuous improvement.

Experience Requirements

• Bachelor's degree in Computer Science, Information Systems, or related discipline with at least five years of related experience, or equivalent training and work experience. Master's degree and Financial Services experience preferred.
• Demonstrated expertise in object-oriented and database technologies leading to enterprise-quality solutions.
• Experience delivering enterprise solutions in iterative or Agile environments.
• Knowledge of software engineering approaches including test automation, build automation, and configuration management frameworks.
• Strong written and verbal technical communication skills.
• Ability to build effective working relationships and improve work product quality.
• Strong organization, thoroughness, and ability to handle competing priorities.
• Ability to maintain focus and develop proficiency in new skills rapidly.
• Experience working in a fast paced environment.
• Experience with Java, Scala, or Python.
• Essential technical skills include AI tool proficiency (e.g., GitHub Copilot, Q Developer, ChatGPT, Claude), strong software development background, and extensive experience with Scrum, Kanban, and continuous improvement.
• Big Data technologies: Hadoop, Spark, Hive, and Trino, with understanding of data skew, PB-scale data, and resource-related job failures.
• Good to have: managing production data pipelines and ETL systems, CI/CD, writing test cases, and AWS certifications.

Education Requirements

• Bachelor's degree in Computer Science, Information Systems, or related discipline required. Master's degree preferred.