Senior Data Engineer

ShadowObjects, LLC; Town of Florida (NY); Remote; USD 120,000 - 140,000; Full time

Job summary

ShadowObjects, LLC seeks a Senior Data Engineer to own data projects end-to-end. The role blends data analysis, engineering, and science responsibilities with a focus on quality, governance, and automation.

You will develop scalable pipelines and dashboards in a DoD environment, collaborating with analysts and stakeholders. The position is fully remote with potential engagement near DoD platforms such as Advana/Jupiter, and requires an active DoD Secret clearance.

Qualifications

• Bachelor’s degree in Data Science, CS, Stats, Eng or related field.
• 5+ years in data engineering & building pipelines/architectures.
• 5+ years in data analysis, data mining, or BI.
• 3+ years using Python/SQL/PySpark in big data environments (e.g., Databricks, Snowflake).
• Strong experience with data visualization tools and advanced Excel.

Responsibilities

• Gather, clean, and analyze complex datasets.
• Design, develop, and maintain ETL/ELT pipelines using Databricks/Delta Lake.
• Implement Medallion Architecture (Bronze-Silver-Gold).
• Manage data ingestion with governance (Collibra).
• Develop dashboards and reports with Tableau/Power BI/Qlik/Foundry.
• Support DoD environments (Advana/Jupiter).
• Mentor team members and drive process improvement.

Skills

Data pipeline design

Databricks

Python

SQL

Data visualization

AWS/Azure/GC

Collibra governance

ETL/ELT

Education

Bachelor's degree in Data Science or related field

Tools

Databricks

Delta Lake

Tableau/Power BI/Qlik/Foundry

Collibra

Job description

Salary Range: $120,000.00 To $140,000.00 Annually

Full Time Professional
Remote
Salary Range:$120,000.00 To $140,000.00 Annually
Job Title: Senior Data Engineer
Clearance Requirement: Active DoD Secret Clearance required
Location: Remote / Pentagon

Position Overview

The Senior Data Engineer is a hybrid role combining Data Analyst, Data Engineer, and Data Scientist responsibilities. This position operates with a high level of autonomy, owning projects end-to-end across functional areas. The ideal candidate is proactive, solution-oriented, and capable of identifying opportunities, reducing complexity, and delivering efficient, high-quality data solutions.

This role also provides guidance to team members, helps remove roadblocks, and contributes to the continuous improvement of processes, systems, and deliverables.

Key Responsibilities

Data Analysis & Insights

• Gather, clean, and analyze complex datasets using advanced analytical techniques
• Apply statistical methods to identify trends, patterns, and anomalies
• Conduct exploratory data analysis (EDA) to support decision-making
• Translate data findings into actionable insights for technical and non-technical stakeholders
• Design, develop, and maintain ETL/ELT pipelines using Databricks and Delta Lake
• Implement Medallion Architecture (Bronze, Silver, Gold) to ensure data quality and scalability
• Manage data ingestion processes with a focus on traceability and governance (e.g., Collibra)
• Build and maintain data pipelines within DoD environments such as Advana/Jupiter
• Troubleshoot, debug, and optimize data pipelines, identifying root causes and implementing solutions
• Develop dashboards, reports, and visualizations using tools such as Tableau, Power BI, Qlik, or Foundry
• Create interactive dashboards and reports to support operational and strategic decisions
• Ensure accuracy, quality, and usability of all data visualizations and deliverables

Integration & System Support

• Collaborate with data engineers, analysts, scientists, and stakeholders to deliver integrated data solutions
• Support real-time data streaming using Databricks Structured Streaming
• Optimize data storage solutions leveraging Delta Lake capabilities (e.g., schema evolution, time travel)

Governance & Compliance

• Maintain data lineage and governance using tools such as Collibra
• Ensure compliance with data security, privacy, and DoD requirements
• Establish and enforce data quality standards across multiple data sources
• Work closely with cross-functional teams to gather and refine requirements
• Communicate findings, recommendations, and metrics clearly and professionally
• Track assigned tasks and provide regular updates to leadership
• Mentor junior team members and support team development initiatives

Required Qualifications

• Bachelor’s degree in Data Science, Computer Science, Statistics, Engineering, or a related field
• 5+ years of experience in data engineering and building data pipelines and architectures
• 5+ years of experience in data analysis, data mining, or business intelligence
• 3+ years of experience developing data solutions using Python, SQL, and/or PySpark in big data environments (e.g., Databricks, Snowflake)
• Strong experience with data visualization tools (Tableau, Power BI, Qlik, Foundry) and advanced Excel (macros, pivot tables, VBA)
• Experience with cloud platforms such as AWS, Azure, or Google Cloud
• Familiarity with big data frameworks (Hadoop, Spark, Kafka)
• Strong understanding of ETL processes, data warehousing, and BI best practices
• Proven ability to analyze large datasets and communicate insights effectively
• Strong problem-solving skills and attention to detail

Preferred Qualifications

• Master’s degree in Data Engineering, Computer Science, or related field
• Experience with DoD data platforms such as Advana and Jupiter
• Familiarity with statistical methods (e.g., hypothesis testing, regression analysis)
• Experience with SAP, Foundry, or advanced analytics platforms
• Experience with NoSQL databases (e.g., Cassandra, MongoDB)
• Knowledge of containerization tools (Docker, Kubernetes)
• Experience with version control systems (Git) and Agile methodologies (Scrum, Kanban)
• Prior experience in a client-facing government environment

Equal Employment Opportunity Statement

ShadowObjects, LLC is an Equal Opportunity Employer. All qualified applicants will receive consideration for employment without regard to race, color, religion, sex (including pregnancy, sexual orientation, and gender identity), national origin, age, disability, genetic information, protected veteran status, or any other characteristic protected by applicable federal, state, or local law.