Principal ETL Developer

Ideal Innovations Incorporated; Bethesda (MD); On-site; USD 150,000 - 190,000; Full time

Job summary

Ideal Innovations Incorporated in Bethesda, MD is seeking an ETL Developer to shape and implement data flow solutions around Apache NiFi. You will design and optimize NiFi pipelines for batch and real-time processing, integrating with AWS, Azure, OCI, Kafka, Elasticsearch, S3, and SQS/SNS.

This role participates in the DevOps lifecycle, mentors junior developers, and collaborates with data architects and engineers to deliver scalable, secure data solutions for critical customer needs.

Qualifications

• Active TS/SCI with polygraph clearance is required.
• Bachelor's degree with 12+ years or Master's with 10+ years, substitutions allowed.
• Deep NiFi data flow design and management in large-scale enterprise settings.
• Proficiency in Java and Python for NiFi processor development.
• Experience with cloud platforms (AWS, Azure, OCI) and data services.
• Strong SQL and NoSQL experience (Postgres, Elasticsearch, DynamoDB).
• Experience with real-time streaming, REST APIs, and data integration.

Responsibilities

• Architect complex NiFi data pipelines for ingestion, transformation, and processing at scale.
• Tune NiFi for batch and real-time performance, memory management, and load balancing.
• Diagnose and resolve NiFi data flow issues and bottlenecks.
• Integrate NiFi with AWS, OCI, Azure and services like Kafka, Elasticsearch, S3, SQS/SNS.
• Set best practices for NiFi development, security, and governance.
• Create comprehensive documentation for NiFi flows, mappings, and architectures.
• Collaborate with data architects, engineers and developers to deliver robust solutions.
• Mentor junior developers and lead peers through code reviews and guidance.

Skills

Mentorship

Cross-functional teamwork

Communication skills

Problem solving

Education

Bachelor's degree

Master's degree

Tools

NiFi

NiFi Registry

NiFi clustering

Java

Python

SQL

REST

AWS

Azure

OCI

Kafka

Elasticsearch

S3

SQS/SNS

PostgreSQL

DynamoDB

Job description

Highlights

• We are looking for an ETL developer to play a pivotal role in shaping, and implementing cutting‑edge data flow solutions centered around Apache NiFi.
• Design and optimize complex NiFi data pipelines, supporting both batch and real‑time processing through performance tuning, memory management, and load balancing.
• Integrate across big data and cloud technologies, connecting NiFi with platforms and services such as AWS, Azure, OCI, Kafka, Elasticsearch, S3, and SQS/SNS.
• Contribute across the full software development lifecycle within a DevOps framework, supporting the design, development, implementation, and testing of customer tools and applications.
• Solve complex data and integration challenges, collaborating with data architects, data engineers, developers, and other stakeholders to deliver robust, scalable, and secure solutions supporting critical customer needs.

Typical Day

The candidate will provide technical expertise and support in the design, development, implementation and testing of customer tools and applications in support of Extracting, Transforming and Loading (ETL) of data into an enterprise Data Lake. The candidate will be responsible for following architectural best practices, and optimizing performance in large‑scale environments, ensuring the delivery of robust, scalable, and secure data flow solutions that drive critical customer needs. Based in a DevOps framework, the ETL Developer participates in the delivery of major projects through all aspects of the software development lifecycle.

Tasks

• Architecting complex NiFi data pipeline design: Design and develop enterprise‑level ETL architectures and implement NiFi data pipelines for large‑scale data ingestion, transformation, and processing from diverse sources.
• Performance optimization and tuning: Optimize NiFi data flows, including processor tuning, memory management, and load balancing, ensuring optimal performance for batch and real‑time processing.
• Advanced troubleshooting and problem resolution: Identify, diagnose, and resolve complex NiFi data flow issues, including performance bottlenecks, data discrepancies, and integration failures.
• Integrating with big data and cloud technologies: Seamlessly integrate NiFi with various databases, big data ecosystems, and cloud platforms (e.g., AWS, OCI, Azure), demonstrating expertise in relevant services (e.g., Kafka, Elasticsearch, S3, SQS/SNS).
• Defining best practices and standards: Establish best practices for NiFi development, deployment, security, and governance, ensuring adherence to enterprise‑wide data management policies.
• Documentation and knowledge sharing: Create and maintain comprehensive documentation for NiFi data flows, mappings, architectures, and standard operating procedures, ensuring knowledge transfer and promoting efficient team operations.
• Collaboration and communication: Collaborate effectively with data architects, data engineers, application/service developers, and other stakeholders to translate business requirements into robust technical solutions and effectively communicate complex technical concepts to both technical and non‑technical audiences.
• Mentorship and team leadership: Mentor junior developers, provide technical guidance, conduct code reviews, and foster a collaborative learning environment.

Required Qualifications

To be considered, must have an active TS/SCI with polygraph security clearance

• Education: Bachelor's degree with 12 or more years of prior relevant experience or Master's degree with 10 or more years of relevant experience. Additional years of experience may be substituted in lieu of a degree.
• The ideal candidate will possess:
  • In‑depth experience designing, developing, and managing complex NiFi data flow solutions in large‑scale enterprise environments.
  • In‑depth knowledge of NiFi architecture, processors, and configurations, along with hands‑on experience with NiFi Registry and clustering for high availability and scalability.
  • Proficiency in programming languages like Java and Python for custom NiFi processor development and scripting for automation.
  • Proficiency writing and optimizing complex queries, along with experience in managing relational and NoSQL databases (e.g., Postgres, Elasticsearch, DynamoDB).
  • Direct experience with real‑time streaming, and API integration (REST) for seamless data connectivity.
  • Direct experience with cloud platforms like AWS, Azure, or OCI and related data services
  • Strong ability to analyze complex data challenges, identify root causes, and implement effective solutions.
  • Strong ability to collaborate effectively with cross‑functional teams, articulate technical concepts clearly, and provide effective mentorship.

Desired Qualifications

• In‑depth experience deploying ETL solutions in an AWS environment.

Position Type Shift Information

• Mon‑Fri, 40 hr work week

Citizenship & Clearance

US Citizenship and TS/SCI with a Full‑Scope Polygraph (FSP) Required

Location

Onsite- Bethesda, MD

Education

• High School (required)
• Masters (preferred)
• Bachelors (preferred)

Experience

• 12 years: Prior relevant experience (required)

Equal Opportunities Statements

Ideal Innovations, Inc. is an Equal Opportunity Employer:

All qualified applicants will receive consideration for employment without regard to race, color, religion, sex, national origin, sexual orientation, gender identity, disability, or veteran status.

Ideal Innovations, Inc. is a VEVRAA Federal Contractor.

Equal Opportunity Employer/Protected Veterans/Individuals with Disabilities

This employer is required to notify all applicants of their rights pursuant to federal employment laws.

For further information, please review the Know Your Rights notice from the Department of Labor.