Senior Data Engineer

Shield Consulting Solutions, Inc.; Maryland; On-site; USD 230,000 - 240,000; Full time

Benefits offered by this job

25 days PTO

11 paid holidays

100% employer-paid healthcare

8% 401(k) employer match

Job summary

Shield Consulting Solutions, Inc. is seeking a Senior Data Engineer to design, build, and optimize analytics data pipelines for scalable, reliable processing.

You will focus on end-to-end data ingress/egress, dataflow design, and Spark-based distributed processing to support complex analytics workloads. You will orchestrate workflows with Apache Airflow, work with SQL databases and big data formats, and ensure data quality and efficient processing across teams.

Qualifications

• Minimum 7+ years of data engineering experience.
• Bachelor’s degree in a technical discipline.
• Proficient with Linux CLI and Linux tools.
• Experience developing Bash scripts to automate processes.
• Proficiency in Python and Java.
• Hands-on Apache Airflow experience: DAG design, scheduling, operators/sensors, orchestration.
• Experience with distributed big data processing engines like Apache Spark.
• Familiarity with SQL databases MySQL, MariaDB, PostgreSQL.
• Experience using Jupyter Notebook.
• Experience with pandas and NumPy for data wrangling.
• Experience with Parquet, JSON, CSV, XML data formats.
• Familiarity with data quality, data validation, and anomaly detection.
• Experience using Git.
• Familiarity with HPC job scheduling tools including Slurm.
• Experience with Jira and Confluence.

Responsibilities

• Design, implement, and optimize data ingress and egress pathways to support analytics processing.
• Apply expertise in dataflow design and data transport mechanisms for dependable pipeline performance.
• Use Apache Spark for distributed processing aligned to analytics workloads.
• Manage workflow orchestration for complex processing using Apache Airflow.

Skills

7+ years of data engineering

Education

Bachelor’s degree in a technical discipline

Tools

Apache Spark

Apache Airflow

Linux CLI

Bash scripting

Python

Java

MySQL

MariaDB

PostgreSQL

Jupyter Notebook

pandas

NumPy

Parquet

JSON

CSV

XML

Git

Slurm

JIRA

Confluence

Job description

This Senior Data Engineer role focuses on building and optimizing analytics data pipelines for efficient, scalable, and reliable processing.

Responsibilities

• Design, implement, and optimize data ingress and egress pathways to support analytics processing
• Apply expertise in dataflow design and data transport mechanisms to ensure dependable pipeline performance
• Use Apache Spark for distributed processing aligned to organizational analytics workloads
• Manage workflow orchestration for complex processing by using Apache Airflow

Requirements

• 7+ years of experience
• Bachelor’s degree in a technical discipline
• Experience using the Linux CLI and Linux tools
• Experience developing Bash scripts to automate manual processes
• Recent software development experience using Python and Java
• Experience with Apache Airflow, including:
  • DAG design
  • Scheduling
  • Operators and sensors
  • Orchestrating, scheduling, and monitoring complex workflows
• Experience with distributed big data processing engines, including Apache Spark
• Familiarity with SQL technologies: MySQL, MariaDB, and PostgreSQL for querying, joining, and aggregating large datasets
• Experience using Jupyter Notebook
• Experience with data wrangling and preprocessing using pandas and NumPy
• Experience working with structured, semi-structured, and unstructured data formats: Parquet, JSON, CSV, XML
• Familiarity with data quality concepts, data validation, and anomaly detection
• Experience using Git source control
• Familiarity with HPC job scheduling tools including Slurm
• Experience with the Atlassian Tool Suite including JIRA and Confluence

Technologies

• Apache Spark
• Apache Airflow
• Linux CLI, Bash
• Python, Java
• MySQL, MariaDB, PostgreSQL
• Jupyter Notebook
• pandas, NumPy
• Parquet, JSON, CSV, XML
• Git
• Slurm
• JIRA, Confluence

Compensation and Location

• Location: Annapolis Junction, MD (onsite)
• Salary: USD 230,000 - 240,000 per year

Benefits

• 25 days PTO
• 11 paid holidays
• 100% employer-paid healthcare for employees and dependents (available day 1)
• 8% 401(k) employer match with immediate vesting