Data Engineering Software Developer (SME)

Amatriot Group, LLC; United States; On-site; USD 235,000 - 265,000; Full time

Job summary

Amatriot Group, LLC is seeking a Data Engineering Software Developer SME to design, build, and maintain scalable data pipelines and platforms in DoD/DISA environments. You will work with Confluent Kafka, Elastic Stack, AWS, and OpenShift Kubernetes to ingest, process, and secure large data sets for analytics and operational dashboards.

The role requires hands-on experience with ETL/ELT, data warehouses and lakes, Linux administration, and DevOps practices.

Qualifications

• Hands on experience designing building and maintaining ETL/ELT data pipelines and integrating multiple disparate data sources.
• Experience with data warehouse and/or data lake technologies such as AWS Redshift, Amazon S3, Hadoop, Snowflake, or similar platforms.
• Experience with Elastic Stack including Elasticsearch, Logstash, and Kibana and/or Kafka for data ingestion and processing.
• Experience with Linux/UNIX system administration and automation.
• Experience with CICD pipelines, containerized pipelines, version control technologies such as Git and Bitbucket and DevOps practices.
• Familiarity with network operations data including logs, metrics, events, ticketing, CMDB data and related data models.

Responsibilities

• Design, build, and maintain scalable data systems and pipelines for analytics and reporting.
• Integrate data from multiple network operations systems into a unified platform.
• Develop Kafka, Elastic, and Logstash pipelines for data ingestion and processing.
• Automate ETL/ELT processes, data validation, cleansing, monitoring.
• Support Kibana visualizations and dashboards using React/JavaScript/HTML.
• Collaborate across software, data analytics, and network operations teams.
• Ensure data security and compliance with DoD/DISA requirements.

Skills

Python

Java

SQL

Shell scripting

Data pipelines

Education

Bachelor's degree in Computer Science

Master's degree in Computer Science

Tools

Confluent Kafka

Elastic Stack

AWS

OpenShift/Kubernetes

Elasticsearch

Logstash

Kibana

Snowflake

Hadoop

Databricks

Job description

Location Washington DC 100 Onsite Clearance Top SecretSCI

Job Type Full time

Target Salary Range 235000 265000

This represents the potential salary range for this position depending on education level years of experience andor certifications in addition to other position specific requirements which may impact salary

Position Overview

The Data Engineering Software Developer SME designs builds integrates and maintains scalable data systems and pipelines that support analytics reporting and operational insight across complex enterprise environments This role serves as a subject matter expert for data ingestion transformation integration platform optimization and secure data engineering within DoD and DISA environments The position develops and supports solutions using Confluent Kafka Elastic Stack AWS Red Hat OpenShiftKubernetes data warehouses and data lakes The role also supports automation CICD observability visualization technical documentation and collaboration across software data analytics and network operations teams

Key Responsibilities

• Data Pipeline Development and Integration
• Design build and maintain scalable and reliable data systems and pipelines to ingest process and transform large complex and disparate data sources for analytics and reporting
• Unify and integrate data from multiple network operations systems into a consistent and accessible platform
• Create and maintain Kafka Elastic and Logstash pipelines supporting enterprise data ingestion and processing
• Automate ETLELT processes data validation cleansing monitoring and related data processing activities
• Data Platforms and Software Development
• Implement data solutions using Confluent Kafka Elastic Stack AWS and Red Hat OpenShiftKubernetes
• Develop and manage data warehouses and data lakes supporting analytics reporting and operational dashboards
• Support Kibana visualizations and dashboards using React JavaScript and HTML
• Apply automation and engineering best practices to improve platform scalability maintainability and reliability
• Performance and Platform Optimization
• Analyze and optimize ingestion throughput query performance indexing strategies and storage efficiency across Elastic and Kafka platforms
• Troubleshoot and resolve issues related to data ingestion processing storage memory utilization partitioning and cluster performance
• Identify and resolve performance bottlenecks affecting data availability and downstream analytics
• Optimize infrastructure for performance scalability and cost effectiveness
• Security Compliance and DevOps
• Ensure data quality accuracy security and compliance with applicable DoD and DISA requirements
• Apply appropriate data security access control and compliance practices
• Support CICD pipelines for automated build test and deployment activities
• Apply version control and DevOps practices to data engineering and software development workflows
• Collaboration and Documentation
• Collaborate with data analysts visualization engineers software developers and other stakeholders to deliver integrated and actionable data solutions
• Document data architecture integration patterns platform configurations and operational procedures
• Develop and maintain technical documentation and runbooks supporting data platforms and pipelines

Qualifications

Education

• Bachelors degree in Computer Science Engineering or a related technical discipline with 12 years of relevant experience or
• Masters degree in Computer Science Engineering or a related technical discipline with 10 years of relevant experience

Experience

• Hands on experience designing building and maintaining ETLELT data pipelines and integrating multiple disparate data sources Required
• Experience with data warehouse andor data lake technologies such as AWS Redshift Amazon S3 Hadoop Snowflake or similar platforms Required
• Experience with Elastic Stack including Elasticsearch Logstash and Kibana andor Kafka for data ingestion and processing Required
• Experience with LinuxUNIX system administration and automation Required
• Experience with CICD pipelines containerized pipelines version control technologies such as Git and Bitbucket and DevOps practices Required
• Familiarity with network operations data including logs metrics events ticketing CMDB data and related data models

Skills

• Proficiency with data engineering tools and languages such as Python Java SQL and shell scripting Required
• Knowledge of data security access controls and compliance frameworks including DoD DISA RMF and STIG requirements Required
• Strong communication and documentation skills supporting collaboration and knowledge sharing Required

Certifications

Current Security certification or higherequivalent certification Required

Clearance

Active TSSCI security clearance Required

Other Requirements

• Willingness and ability to complete and maintain additional customer suitability screenings throughout employment Required

Preferred Qualifications

• Experience supporting DISA DISN or other DoD network environments
• Experience developing and deploying software applications that meet DoD security standards including applicable STIGs
• Knowledge of network security encryption and access controls within classified environments
• Experience with big data technologies and platforms such as Kafka Spark NiFi Hadoop Elasticsearch Logstash Databricks and ELK Stack
• Experience applying big data technologies to text mining summarization search and entity extraction
• Experience with cloud based data platforms including AWS Azure GCP or AWS GovCloud
• Experience with cloud integrated platforms and Infrastructure as Code including networking and security policies
• Familiarity with Kubernetes deployment platform upgrades and patching
• Experience with configuration management technologies such as Ansible Puppet or Chef
• Experience with monitoring alerting and observability technologies such as Prometheus Grafana and Elastic
• Experience with business intelligence and visualization tools such as Kibana
• Experience developing and maintaining technical documentation and operational runbooks
• Strong understanding and practical experience with Agile methodologies including Scrum and SAFe
• Experience with Atlassian Jira and Confluence
• Confluent Developer certification
• Elastic Certified Engineer certification
• Experience working remotely with geographically dispersed teams and within matrixed organizations
• Experience combining software development integration and data engineering practices
• Experience applying knowledge of system architecture networking and centralized logging using ELK to support data transformation initiatives