Senior Data Engineer: Scalable Pipelines & Spark Mastery

Shield Consulting Solutions, Inc.; Maryland; On-site; USD 230,000 - 240,000; Full time

Benefits offered by this job

25 days PTO

11 paid holidays

100% employer-paid healthcare

8% 401(k) employer match

Job summary

Shield Consulting Solutions, Inc. is seeking a Senior Data Engineer to design, build, and optimize analytics data pipelines for scalable, reliable processing.

You will focus on end-to-end data ingress/egress, dataflow design, and Spark-based distributed processing to support complex analytics workloads. You will orchestrate workflows with Apache Airflow, work with SQL databases and big data formats, and ensure data quality and efficient processing across teams.

Qualifications

• Minimum 7+ years of data engineering experience.
• Bachelor’s degree in a technical discipline.
• Proficient with Linux CLI and Linux tools.
• Experience developing Bash scripts to automate processes.
• Proficiency in Python and Java.
• Hands-on Apache Airflow experience: DAG design, scheduling, operators/sensors, orchestration.
• Experience with distributed big data processing engines like Apache Spark.
• Familiarity with SQL databases MySQL, MariaDB, PostgreSQL.
• Experience using Jupyter Notebook.
• Experience with pandas and NumPy for data wrangling.
• Experience with Parquet, JSON, CSV, XML data formats.
• Familiarity with data quality, data validation, and anomaly detection.
• Experience using Git.
• Familiarity with HPC job scheduling tools including Slurm.
• Experience with Jira and Confluence.

Responsibilities

• Design, implement, and optimize data ingress and egress pathways to support analytics processing.
• Apply expertise in dataflow design and data transport mechanisms for dependable pipeline performance.
• Use Apache Spark for distributed processing aligned to analytics workloads.
• Manage workflow orchestration for complex processing using Apache Airflow.

Skills

7+ years of data engineering

Education

Bachelor’s degree in a technical discipline

Tools

Apache Spark

Apache Airflow

Linux CLI

Bash scripting

Python

Java

MySQL

MariaDB

PostgreSQL

Jupyter Notebook

pandas

NumPy

Parquet

JSON

CSV

XML

Git

Slurm

JIRA

Confluence

Job description

Shield Consulting Solutions, Inc. is seeking a Senior Data Engineer to design, build, and optimize analytics data pipelines for scalable, reliable processing.

You will focus on end-to-end data ingress/egress, dataflow design, and Spark-based distributed processing to support complex analytics workloads. You will orchestrate workflows with Apache Airflow, work with SQL databases and big data formats, and ensure data quality and efficient processing across teams.