ETL Developer

Ideal Innovations, Inc.

United States

On-site

USD 90,000 - 140,000

Full time

14 days+

Get a reply from this employer

Your resume and cover letter, tailored to exactly what they’re hiring for.

Get past ATS filters

Job summary

Ideal Innovations, Inc. seeks an ETL Developer specializing in Apache NiFi to design and implement enterprise data pipelines for batch and real-time processing. You will optimize performance, memory usage, and load balancing across big data and cloud platforms.

Responsibilities include integrating NiFi with AWS/Azure/OCI, Kafka, Elasticsearch, S3, and SQS/SNS, and collaborating with data architects and engineers within a DevOps framework to deliver robust, secure data flow solutions.

Qualifications

  • Active TS/SCI with polygraph clearance required.
  • Bachelor's degree with 4+ years or Master's with 2+ years of relevant experience.
  • Experience designing, developing, managing NiFi data flows in large-scale environments.
  • Experience Java and/or Python and automation scripting.
  • Experience with relational and NoSQL databases (Postgres, Elasticsearch, DynamoDB).

Responsibilities

  • Build complex NiFi data pipelines for large-scale ingestion, transformation, and processing.
  • Tune NiFi performance including memory management and load balancing.
  • Integrate NiFi with AWS, OCI, Azure and cloud services (Kafka, Elasticsearch, S3, SQS/SNS).
  • Follow best practices for NiFi development, deployment, security, and governance.
  • Document data flows, architectures, and operating procedures; share knowledge.

Skills

TS/SCI clearance
Java
Python
REST API
NoSQL DB
Elasticsearch
DynamoDB
AWS
Azure
OCI
NiFi design
Performance tuning

Education

Bachelor's degree
Master's degree

Tools

NiFi
NiFi Registry
NiFi clustering

Job description

Highlights

We are looking for an ETL developer to play a pivotal role in shaping, and implementing cutting-edge data flow solutions centered around Apache NiFi. Design and optimize complex NiFi data pipelines, supporting both batch and real-time processing through performance tuning, memory management, and load balancing. Integrate across big data and cloud technologies, connecting NiFi with platforms and services such as AWS, Azure, OCI, Kafka, Elasticsearch, S3, and SQS/SNS. Contribute across the full software development lifecycle within a DevOps framework, supporting the design, development, implementation, and testing of customer tools and applications. Solve complex data and integration challenges, collaborating with data architects, data engineers, developers, and other stakeholders to deliver robust, scalable, and secure solutions supporting critical customer needs.

Typical Day

The candidate will provide technical support in the design, development, implementation and testing of customer tools and applications in support of Extracting, Transforming and Loading (ETL) of data into an enterprise Data Lake. The candidate will be responsible for following architectural best practices, and optimizing performance in large-scale environments, ensuring the delivery of robust, scalable, and secure data flow solutions that drive critical customer needs. Based in a DevOps framework, the ETL Developer participates in the delivery of major projects through all aspects of the software development lifecycle.

Tasks

Building complex NiFi data pipeline design: Develop and implement enterprise-level ETL NiFi data pipelines for large-scale data ingestion, transformation, and processing from diverse sources. Performance optimization and tuning: Optimize NiFi data flows, including processor tuning, memory management, and load balancing, ensuring optimal performance for batch and real-time processing. Troubleshooting and problem resolution: Identify, diagnose, and resolve complex NiFi data flow issues, including performance bottlenecks, data discrepancies, and integration failures. Integrating with big data and cloud technologies: Seamlessly integrate NiFi with various databases, big data ecosystems, and cloud platforms (e.g., AWS, OCI, Azure), demonstrating familiarity in relevant services (e.g., Kafka, Elasticsearch, S3, SQS/SNS). Follow best practices and standards: Follow establish best practices for NiFi development, deployment, security, and governance, ensuring adherence to enterprise-wide data management policies. Documentation and knowledge sharing: Create and maintain comprehensive documentation for NiFi data flows, mappings, architectures, and standard operating procedures, ensuring knowledge transfer and promoting efficient team operations. Collaboration and communication: Collaborate effectively with data architects, data engineers, application/service developers, and other stakeholders to translate business requirements into robust technical solutions.

Required Qualifications

To be considered, must have an active TS/SCI with polygraph security clearance.

Education: Bachelor's degree with 4 or more years of prior relevant experience or Master's degree with 2 or more years of relevant experience. Will consider experience in lieu of a degree. The ideal candidate will possess: Direct experience designing, developing, and managing complex NiFi data flow solutions in large-scale enterprise environments. Experience in programming languages like Java and/or Python and scripting for automation. Experience writing and optimizing complex queries, along with experience in interacting with relational and NoSQL databases (e.g., Postgres, Elasticsearch, DynamoDB). Experience with real-time streaming, and API integration (REST) for seamless data connectivity. Experience with cloud platforms like AWS, Azure, or OCI and related data services. Ability to analyze complex data challenges, identify root causes, and implement effective solutions. Ability to collaborate effectively with cross-functional teams and articulate technical concepts clearly.

Desired Qualifications

Experience deploying ETL solutions in an AWS environment.

Experience developing custom NiFi processors with Java and/or Python.

Direct knowledge of NiFi architecture, processors, and configurations, along with hands-on experience with NiFi Registry and clustering for high availability and scalability.

Position Type Shift Information

Mon-Fri, 40 hr work week.

Citizenship & Clearance

US Citizenship and TS/SCI with a Full-Scope Polygraph (FSP) Required.

Location

Onsite- Bethesda, MD.

Equal Opportunity

Ideal Innovations, Inc. is an Equal Opportunity Employer: All qualified applicants will receive consideration for employment without regard to race, color, religion, sex, national origin, sexual orientation, gender identity, disability, or veteran status.

Ideal Innovations, Inc. is a VEVRAA Federal Contractor.

Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Principal ETL Developer
Principal ETL Developer

Ideal Innovations, Inc. • United States

On-site
USD 140,000 - 190,000
ETL Developer
ETL Developer

Ideal Innovations, Inc. • Bethesda (MD)

On-site
USD 87,000 - 134,000
ETL Developer
ETL Developer

Ideal Innovations, Inc. • Bethesda (MD)

On-site
USD 87,000 - 134,000
Principal ETL Developer
Principal ETL Developer

Ideal Innovations, Inc. • Bethesda (MD)

On-site
USD 131,000 - 202,000
Principal ETL Developer
Principal ETL Developer

Ideal Innovations, Inc. • Bethesda (MD)

On-site
USD 131,000 - 202,000
Principal ETL Developer
Principal ETL Developer

Leidos • Bethesda (MD)

On-site
USD 126,100 - 227,950
Paid Time Off
401K with 6% company match
Flexible Schedules
+2
ETL Java Developer
ETL Java Developer

Leidos • Bethesda (MD)

On-site
USD 69,550 - 125,725
Senior NiFi ETL Architect (TS/SCI)
Senior NiFi ETL Architect (TS/SCI)

Ideal Innovations, Inc. • United States

On-site
USD 140,000 - 190,000
ETL Developer
ETL Developer

Leidos • Bethesda (MD)

On-site
USD 94,000 - 157,000
Paid Time Off
11 paid Holidays
401K with 6% company match
ETL Java Developer
ETL Java Developer

Leidos • United States

On-site
USD 69,550 - 125,725
Paid Time Off
401(k) with company match
Flexible Schedules
+1