Senior Dataflow Engineer - Spark, Airflow, Big Data

Erias Ventures, LLC.; Maryland; On-site; USD 229,000 - 251,000; Full time

Benefits offered by this job

Above Market Pay

401k Immediate Vesting

Spot Bonuses

Professional Development Bonuses

Medical Coverage

Vision & Dental

Disability Insurance

Wellness Reimbursement

Paid Time Off

Birthday Off

Internet Reimbursement

Equipment Allowance

Company Events

Job summary

Erias Ventures, LLC is seeking a Software Engineer with expertise in dataflow design and Spark-based processing to design, implement, and optimize analytics pipelines. The role demands a Top-Secret/SCI clearance with polygraph and a strong background in Python and Java development.

You will architect and optimize data ingress/egress pathways, leverage Airflow for complex workflows, and work with SQL-based data stores and large datasets to deliver scalable solutions for mission-focused analytics.

Qualifications

• Extensive experience with Linux CLI and scripting for automation.
• Python and Java software development experience.
• Experience with Airflow for orchestration of complex workflows.
• Big data processing with Apache Spark.
• Proficient SQL querying with MySQL, MariaDB, PostgreSQL.
• Data wrangling with Pandas/NumPy and diverse data formats.
• Git source control and Jupyter Notebook usage.
• Awareness of data quality concepts and anomaly detection.

Responsibilities

• Design, implement, and optimize data ingress/egress pathways for analytics workloads.
• Develop and maintain Spark-based distributed processing pipelines.
• Create and monitor workflows with Apache Airflow.
• Collaborate across disciplines to deliver scalable data solutions.
• Ensure secure, efficient data processing in support of mission needs.

Skills

Linux CLI

Bash scripting

Python

Java

Apache Airflow

Apache Spark

SQL Databases

Jupyter Notebook

Pandas/NumPy

Parquet/JSON/CSV/XML

Data quality

Git

Education

Master's degree in Computer Science or related

Bachelor's degree in Computer Science or related

9 years SWE experience (alternative)

Tools

Slurm

JIRA

Confluence

Job description

Erias Ventures, LLC is seeking a Software Engineer with expertise in dataflow design and Spark-based processing to design, implement, and optimize analytics pipelines. The role demands a Top-Secret/SCI clearance with polygraph and a strong background in Python and Java development.

You will architect and optimize data ingress/egress pathways, leverage Airflow for complex workflows, and work with SQL-based data stores and large datasets to deliver scalable solutions for mission-focused analytics.