Senior Data Engineer

Imagine Believe Realize, LLC; Rockledge (FL); Remote; USD 90,000 - 130,000; Full time

Job summary

Join a forward-thinking company as a Senior Data Engineer, where you will play a pivotal role in developing next-generation data collection systems. This exciting position involves building robust and scalable enterprise web portal solutions hosted in AWS, working closely with a dynamic team to enhance data architectures and optimize data processes. If you are passionate about leveraging your extensive experience in data engineering and want to contribute to innovative projects, this opportunity is perfect for you. Embrace the chance to work in a collaborative environment that values continuous learning and professional growth.

Qualifications

• 10+ years of IT experience focusing on enterprise data architecture and management.
• Experience with Databricks and advanced SQL required.

Responsibilities

• Plan and maintain data architectures aligned with business requirements.
• Create and manage ETL/ELT pipelines and data transformations.

Skills

Enterprise Data Architecture

Databricks

Data Modeling

ETL Tools (SSIS, Pentaho)

Advanced SQL

AWS Environment

Python

Data Quality Validation

Education

Bachelor’s in IT related field

Tools

AWS

CI/CD Pipelines

Spark

Kafka

Great Expectations

Job description

The Senior Data Engineer must be able to meet the key criteria below:

1. Location: 100% telework
2. Years' Experience: 10+ years
3. Education: Bachelor’s in IT related field
4. Security Clearance: IBR is a federal contractor. Applicants must be able to meet the requirements to obtain a Public Trust security clearance. NOTE: United States Citizenship is required to be eligible to obtain this security clearance.
5. Work Authorization: Must show that applicant is legally permitted to work in the United States.
6. Employment Type: Full-Time, W-2
7. Key Skills:
   • 10+ years of IT experience focusing on enterprise data architecture and management
   • Experience with Databricks required
   • 8+ years experience in Conceptual/Logical/Physical Data Modeling & expertise in Relational and Dimensional Data Modeling
   • Experience with Great Expectations or other data quality validation frameworks
   • Experience with ETL and ELT tools such as SSIS, Pentaho, and/or Data Migration Services
   • Advanced level SQL experience (Joins, Aggregation, Windowing functions, Common Table Expressions, RDBMS schema design, Postgres performance optimization)
   • Experience with AWS environment, CI/CD pipelines, and Python (Python 3) a bonus

Overview

Do you want to help build a portfolio of next-generation mobile-enabled data collection systems and enterprise portals? As a Data Engineer at IBR, you will support the Agile based engineering of robust, secure, and scalable enterprise web portal solutions hosted in AWS. This position will work closely with the solutions delivery team to support the operations team performing Deployment, Systems Integration Testing, and Operations & Maintenance activities.

Responsibilities

• Plan, create, and maintain data architectures, ensuring alignment with business requirements
• Obtain data, formulate dataset processes, and store optimized data
• Identify problems and inefficiencies and apply solutions
• Determine tasks where manual participation can be eliminated with automation
• Identify and optimize data bottlenecks, leveraging automation where possible
• Create and manage data lifecycle policies (retention, backups/restore, etc)
• In-depth knowledge for creating, maintaining, and managing ETL/ELT pipelines
• Create, maintain, and manage data transformations
• Maintain/update documentation
• Create, maintain, and manage data pipeline schedules
• Monitor data pipelines
• Create, maintain, and manage data quality gates (Great Expectations) to ensure high data quality
• Support AI/ML teams with optimizing feature engineering code
• Expertise in Spark/Python/Databricks, Data Lake and SQL
• Create, maintain, and manage Spark Structured Streaming jobs, including using the newer Delta Live Tables and/or DBT
• Research existing data in the data lake to determine best sources for data
• Create, manage, and maintain ksqlDB and Kafka Streams queries/code
• Data driven testing for data quality
• Maintain and update Python-based data processing scripts executed on AWS Lambdas
• Unit tests for all the Spark, Python data processing and Lambda codes
• Maintain PCIS Reporting Database data lake with optimizations and maintenance (performance tuning, etc)
• Streamlining data processing experience including formalizing concepts of how to handle lake data, defining windows, and how window definitions impact data freshness

Qualifications

• 10+ years of IT experience focusing on enterprise data architecture and management
• Experience in Conceptual/Logical/Physical Data Modeling & expertise in Relational and Dimensional Data Modeling
• Experience with Databricks, Structured Streaming, Delta Lake concepts, and Delta Live Tables required
• Experience leading and architecting enterprise-wide initiatives specifically system integration, data migration, transformation, data warehouse build, data mart build, and data lakes implementation / support
• Advanced level SQL experience (Joins, Aggregation, Windowing functions, Common Table Expressions, RDBMS schema design, Postgres performance optimization)
• Debug, troubleshoot, design and implement solutions to complex technical issues
• Experience with large-scale, high-performance enterprise big data application deployment and solution
• Understanding how to create DAGs to define workflows
• Familiarity with CI/CD pipelines, containerization, and pipeline orchestration tools such as Airflow, Prefect, etc a bonus but not required
• Architecture experience in AWS environment a bonus
• Knowledge of Python (Python 3 desired) for CI/CD pipelines a bonus
• Experience working with JSON and defining JSON Schemas a bonus
• Ability to thrive in a team-based environment
• Experience briefing the benefits and constraints of technology solutions to technology partners, stakeholders, team members, and senior level of management

Physical Demands

Position consists of sitting for long periods of time, bending, stooping, crouching, and lifting up to 20 pounds. Frequently uses hands/fingers for manipulation of keyboard and mouse.

Work Environment

Work is performed primarily indoors in a well-lit office environment. The environment is normally air conditioned, but conditions may change dependent upon circumstances. Work may need to be performed in a fast-paced environment requiring quick thinking and rapid judgments.

About IBR

Imagine Believe Realize, LLC (IBR) is an emerging small business focused on delivering software and systems engineering solutions to government and commercial clients. Our talent acquisition strategy is tailored to career seeking candidates who embrace continuous learning and desire to grow as a professional in the software/systems engineering industry.

Upon hire and in compliance with federal law, all persons hired are required to verify identity and eligibility to work in the United States, and to complete the required employment eligibility verification and background check. IBR is a Federal Contractor.

Imagine Believe Realize, LLC is proud to be an Equal Opportunity and Affirmative Action Employer. We do not discriminate based upon race, age, religion, color, national origin, gender (including pregnancy, childbirth, or related medical conditions), sexual orientation, gender identity, gender expression, status as a protected veteran, status as an individual with a disability, or other applicable legally protected characteristics.