Big Data Engineer - RQ308

ManpowerGroup; McLean (VA); On-site; USD 120,000 - 150,000; Full time

Job summary

ManpowerGroup is seeking a highly skilled Big Data Engineer to design, develop, and optimize large-scale data processing systems. You will design pipelines, implement data integration, and ensure performance, scalability, and reliability of big data platforms.

You will collaborate with cross-functional teams and stay current with technologies like Hadoop, Spark, and Python/Scala. The ideal candidate has five years of experience and strong OO/database concepts, plus testing/CI/CD experience.

Qualifications

• Bachelor's degree in Computer Science or related field with 5+ years of experience.
• Master's degree and Financial Services experience preferred.
• Strong OO and database concepts; enterprise-grade solutions.
• Experience in Agile development environments.
• Experience with test automation and build/config management.
• Excellent written and verbal technical communication.

Responsibilities

• Design, develop, and maintain large-scale data processing pipelines.
• Ingest, store, transform, and analyze scalable data solutions.
• Collaborate with cross-functional teams to translate requirements.
• Optimize pipelines for performance and reliability.
• Develop automated tests and data quality assurance.
• Troubleshoot production pipelines and resolve issues.
• Work with data scientists and analysts to enable data-driven decisions.

Skills

Distributed systems

Cloud platforms

Hadoop

Spark

Python/Scala

SQL (Window functions)

Spark tuning

Data quality testing

Agile/Scrum

Education

Bachelor's in CS or related

Master's degree (pref.)

Tools

Hadoop

Spark

Hive

Trino

Python

Scala

SQL

Job description

We are seeking a highly skilled and experienced Big Data Engineer to design, develop, and optimize large-scale data processing systems.

In this role, you will work closely with cross-functional teams to architect data pipelines, implement data integration solutions, and ensure the performance, scalability, and reliability of big data platforms.

The ideal candidate will have deep expertise in distributed systems, cloud platforms, and modern big data technologies such as Hadoop, Spark etc.

Responsibilities:

• Design, develop, and maintain large-scale data processing pipelines using Big Data technologies (e.g., Hadoop, Spark, Python, Scala).

• Implement data ingestion, storage, transformation, and analysis of solutions that are scalable, efficient, and reliable.

• Stay current with industry trends and emerging Big Data technologies to continuously improve the data architecture

• Collaborate with cross-functional teams to understand business requirements and translate them into technical solutions.

• Optimize and enhance existing data pipelines for performance, scalability, and reliability.

• Develop automated testing frameworks and implement continuous testing for data quality assurance.

• Conduct unit, integration, and system testing to ensure the robustness and accuracy of data pipelines.

• Work with data scientists and analysts to support data-driven decision-making across the organization.

• Ability to write and maintain automated unit, integration, and end-to-end tests

• Monitor and troubleshoot data pipelines in production environments to identify and resolve issues.

Education/Experience Requirements:

• Bachelor's degree in Computer Science, Information Systems or related discipline with at least five (5) years of related experience, or equivalent training and/or work experience; Master's degree and past Financial Services industry experience preferred.

• Demonstrated technical expertise in Object Oriented and database technologies/concepts which resulted in deployment of enterprise quality solutions.

• Past experience with developing enterprise quality solutions in an iterative or Agile environment.

• Extensive knowledge of industry leading software engineering approaches including Test Automation, Build Automation and Configuration Management frameworks.

• Strong written and verbal technical communication skills.

• Demonstrated ability to develop effective working relationships that improved the quality of work products.

• Should be well organized, thorough, and able to handle competing priorities.

• Ability to maintain focus and develop proficiency in new skills rapidly.

• Ability to work in a fast paced environment.

• Experience with object oriented programming languages such as Java, Scala or Python.

Essential Technical Skills:

• AI Tool Proficiency: Hands-on experience with AI development tools (GitHub Copilot, Q Developer, ChatGPT, Claude, etc.)

• Technical Background: Strong software development background with ability to contribute to technical discussions

• Agile Methodology: Extensive experience with Scrum, Kanban, and continuous improvement practices

• Big Data technologies:

• Experience with Big data technologies such as Hadoop, Spark, Hive & Trino

• Understanding of common issues like:

• Data skew and strategies to mitigate it.

• Working with massive data volumes in PetaBytes.

• Troubleshooting job failures due to resource limitations, bad data, scalability challenged.

• Real-world debugging and mitigation experience.

AI Skills:

• Prompt Engineering: Proficiency in crafting effective prompts for AI coding assistants and analysis tools

• AI Workflow Design: Experience redesigning development processes to leverage AI capabilities

• Data Analysis: Ability to interpret AI-generated insights and translate them into actionable team improvements

• Change Management: Experience leading teams through AI adoption and workflow transformation

SQL Skills (Window Functions, Joins, Complex Queries):

• SQL window functions, multi-table joins, aggregations.

• Write/optimize SQL queries on the spot.

• Experience handling edge cases like NULLs, duplicates, ordering, etc.

Apache Spark (Development, Internals & Tuning):

• Understanding of Sparks core architecture - executors, tasks, stages, DAG.

• Spark performance tuning techniques: partitioning, caching, broadcast joins, etc.

• Troubleshooting slow running/stuck jobs or resource issues in Spark.

• Experience optimizing Spark jobs for large-scale datasets.

Cloud Technologies:

• Exposure to AWS services like S3, EMR, Glue, Lambda, Athena, etc. (Answer: how have you used S3 with Spark? (e.g., dealing with file formats, consistency issues)).

• EKS, Serverless knowledge, etc.

Programming - Python or Scala:

• Ability to write clean, modular, and performant code.

• Experience in functional programming concepts (e.g., immutability, higher-order functions).

• Give real-world use cases where you've written scalable data processing code.

• Understanding of collections, concurrency, and memory management.

Good to have:

• Experience with managing production data pipelines/ETL systems

• Experience with CI/CD

• Experience writing test cases

• AWS certifications

ManpowerGroup is committed to providing equal employment opportunities in a professional, high quality work environment. It is the policy of ManpowerGroup and all of its subsidiaries to recruit, train, promote, transfer, pay and take all employment actions without regard to an employee's race, color, national origin, ancestry, sex, sexual orientation, gender identity, genetic information, religion, age, disability, protected veteran status, or any other basis protected by applicable law.