Data Engineering Software Developer Sr.

The Amatriot Group; Washington (District of Columbia); In loco; USD 175.000 - 200.000; Tempo pieno

Descrizione del lavoro

The Amatriot Group in Washington, DC is seeking a Senior Data Engineer to design, build, and sustain scalable data systems and pipelines for analytics and operations. You will ingest, process, and unify large data sources while ensuring security, DoD/DISA compliance, and data quality.

The role uses Confluent Kafka, Elastic Stack, AWS, and OpenShift/Kubernetes with data warehouses and lakes to deliver reliable data solutions, automation, CI/CD, and dashboards.

Competenze

• This role requires 8+ years of data engineering experience.
• Experience designing and maintaining ETL/ELT pipelines.
• Proficiency with data warehouses and data lakes (e.g., Redshift, S3, Snowflake).
• Hands-on with Elastic Stack and Kafka for ingestion and processing.
• Linux/UNIX systems administration and automation.
• CI/CD pipelines and DevOps practices.

Mansioni

• Design, build, and maintain scalable data systems and pipelines for ingesting, processing, and transforming large data sources.
• Develop and manage data warehouses and data lakes for analytics and dashboards.
• Analyze and optimize throughput, performance, and storage across Elastic and Kafka platforms.
• Support Kibana visualizations and dashboards; collaborate with analysts and visualization engineers.
• Document architecture, runbooks, and collaborate across teams.

Conoscenze

Python

Java

SQL

Shell scripting

Kafka

Elasticsearch

Kubernetes

Formazione

Bachelor's degree in Computer Science or related field

Strumenti

AWS

OpenShift

Logstash

Kibana

Git

Descrizione del lavoro

Location: Washington, DC 100% Onsite

Clearance: Top Secret/SCI

Job Type: Full-time

Target Salary Range*: $175,000 - $200,000

• This represents the potential salary range for this position depending on education level, years of experience and/or certifications in addition to other position specific requirements which may impact salary

Position Overview

The Senior Data Engineer designs, builds, integrates, and sustains scalable data systems and pipelines that support analytics, reporting, and operational insight across complex enterprise environments. This role focuses on ingesting, processing, transforming, and unifying large and disparate data sources while ensuring data quality, security, performance, and compliance with DoD and DISA requirements.

The Senior Data Engineer works with Confluent Kafka, Elastic Stack, AWS, Red Hat OpenShift/Kubernetes, data warehouses, and data lakes to deliver reliable and efficient data solutions. The role also supports automation, CI/CD, platform optimization, visualization capabilities, and collaboration across data, software, and network operations teams.

Key Responsibilities

Data Pipeline Engineering and Integration

• Design, build, and maintain scalable and reliable data systems and pipelines for ingesting, processing, and transforming large, complex, and disparate data sources.
• Unify and integrate data from multiple network operations systems into a consistent and accessible platform.
• Create and maintain Kafka, Elastic, and Logstash pipelines to support enterprise data ingestion and processing.
• Automate ETL/ELT processes, data validation, cleansing, monitoring, and related data-engineering activities.

Data Platforms and Architecture

• Implement data solutions using Confluent Kafka, Elastic Stack, AWS, and Red Hat OpenShift/Kubernetes.
• Develop and manage data warehouses and data lakes supporting analytics, reporting, and operational dashboards.
• Optimize data infrastructure for scalability, reliability, performance, and cost effectiveness.
• Apply automation and engineering best practices to improve platform maintainability and operational efficiency.

Performance and Platform Optimization

• Analyze and optimize ingestion throughput, query performance, indexing strategies, storage efficiency, and cluster utilization across Elastic and Kafka platforms.
• Troubleshoot and resolve issues related to data ingestion, processing, storage, memory, partitioning, and cluster performance.
• Identify and resolve platform bottlenecks affecting data availability and downstream analytics.

Visualization and Analytics Support

• Support Kibana visualizations and dashboards.
• Support visualization capabilities using React, JavaScript, and HTML.
• Collaborate with data analysts, visualization engineers, and other stakeholders to deliver integrated and actionable data solutions.

Security, Compliance, and DevOps

• Ensure data quality, accuracy, security, and compliance with applicable DoD and DISA requirements.
• Support CI/CD pipelines for automated build, test, and deployment activities.
• Apply version control and DevOps practices to data-engineering workflows.
• Support secure operation of data platforms through appropriate access controls and configuration standards.

Documentation and Collaboration

• Document data architecture, integration patterns, platform configurations, and operational procedures.
• Maintain technical documentation and runbooks supporting data platforms and pipelines.
• Collaborate across software, integration, data, analytics, and network operations teams to support enterprise data initiatives.

Qualifications

Education

• Bachelor’s degree in Computer Science, Engineering, or a related technical discipline with 8+ years of relevant experience. [Required]

Experience

• Hands-on experience designing, building, and maintaining ETL/ELT data pipelines and integrating multiple disparate data sources. [Required]
• Experience with data warehouse and/or data lake technologies such as AWS Redshift, Amazon S3, Hadoop, Snowflake, or similar platforms. [Required]
• Experience with Elastic Stack, including Elasticsearch, Logstash, and Kibana, and/or Kafka for data ingestion and processing. [Required]
• Experience with Linux/UNIX system administration and automation. [Required]
• Experience with CI/CD pipelines, containerized pipelines, version control technologies such as Git and Bitbucket, and DevOps practices. [Required]
• Familiarity with network operations data, including logs, metrics, events, ticketing, CMDB data, and related data models.

Skills

• Proficiency with data engineering technologies and languages such as Python, Java, SQL, and shell scripting. [Required]
• Knowledge of data security, access controls, and compliance frameworks, including DoD, DISA, RMF, and STIG requirements. [Required]
• Strong communication and documentation skills supporting collaboration and knowledge sharing. [Required]

Certifications

• Current Security+ certification or higher/equivalent certification. [Required]

Clearance

• Active TS/SCI security clearance. [Required]

Other Requirements

• Willingness and ability to complete and maintain additional customer suitability screenings throughout employment. [Required]

Preferred Qualifications

• Experience supporting DISA, DISN, or other DoD network environments.
• Experience developing and deploying software applications that meet DoD security standards, including applicable STIGs.
• Knowledge of network security, encryption, and access controls within classified environments.
• Experience with big data technologies and platforms such as Kafka, Spark, NiFi, Hadoop, Elasticsearch, Logstash, Databricks, and ELK Stack.
• Experience applying big data technologies to text mining, summarization, search, and entity extraction.
• Experience with cloud-based data platforms, including AWS, Azure, GCP, or AWS GovCloud.
• Experience with cloud-integrated platforms and Infrastructure as Code, including networking and security policies.
• Familiarity with Kubernetes deployments, platform upgrades, and patching.
• Experience with configuration management technologies such as Ansible, Puppet, or Chef.
• Experience with monitoring, alerting, and observability technologies such as Prometheus, Grafana, and Elastic.
• Experience with business intelligence and visualization tools such as Kibana.
• Experience developing and maintaining technical documentation and operational runbooks.
• Strong understanding and practical experience with Agile methodologies, including Scrum and SAFe.
• Experience with Atlassian Jira and Confluence.
• Confluent Developer certification.
• Elastic Certified Engineer certification.
• Experience working remotely with geographically dispersed teams and within matrixed organizations.
• Experience combining software, integration, and data engineering practices.
• Experience applying knowledge of system architecture, networking, and centralized logging using ELK to support data transformation initiatives.