Spark Data Engineer

Bespoke Technologies; Tysons (VA); On-site; USD 120,000 - 170,000; Full time

Job summary

Bespoke Technologies in Tysons, VA seeks a Spark Data Engineer to design and optimize data pipelines. The role focuses on building scalable data processing using Spark, Python, and AWS services in a cloud environment.

The engineer will work with SQL-based data stores and Linux environments, writing robust scripts and unit tests to ensure data quality and reliability.

Qualifications

• Experience building and managing data pipelines.
• Experience with Python programming.
• Experience with cloud computing using AWS services.
• Experience processing data with Apache Spark.
• Experience with an RDBMS (PostgreSQL, Oracle, MySQL) and writing SQL queries.
• Experience using Linux and shell scripting.
• Experience analyzing data in formats like CSV, XML, JSON, Avro, Parquet.
• Experience writing and validating unit tests.

Skills

Data pipelines

Python

AWS

Apache Spark

SQL

Linux

Shell scripting

Data formats (CSV, XML, JSON, Parquet)

Unit testing

Job description

BT-454 – Spark Data Engineer

Location: Tysons, VA

Required skills

• Demonstrates experience building and managing data pipelines
• Demonstrates experience with Python
• Demonstrates experience with cloud computing, using AWS services
• Demonstrates experience processing data using Apache Spark
• Demonstrates experience with an RDBMS (PostgreSQL, Oracle, MySQL) and writing SQL queries
• Demonstrates experience with Linux and shell scripting
• Demonstrates experience analyzing data in different file formats like CSV, XML, JSON, Avro, Parquet, etc.
• Demonstrates experience writing and validating unit tests

Highly desired:

• Experience with NiFi, Apache AirFlow, or an equivalent solution or tool for orchestrating data pipelines
• Experience with Java or Scala
• Experience administering an EMR/Spark cluster
• Experience conducting performance tuning of a Spark job
• Experience developing cloud-based security solutions
• Experience following a configuration management process to review and deploy code as part of releases