Senior Data Engineer - Data Lake & Spark Architect (Flex PTO)

Pyramid Systems, Inc.; United States; On-site; USD 106,131 - 159,196; Full time

Benefits offered by this job

Employee Stock Ownership Program

FlexPTO

Learning and development opportunities

Job summary

Pyramid Systems, Inc. is seeking a Senior Data Engineer to create innovative architecture solutions. Applicants should have at least 8 years of IT experience with enterprise data architecture and management, including expertise in Databricks, ETL/ELT tools, and advanced SQL. Responsibilities include managing data pipelines, supporting AI/ML teams, and ensuring data quality. The position offers a competitive salary of USD $106,131 - $159,196, along with benefits such as an Employee Stock Ownership Program and a flexible workplace approach.

Qualifications

• 8+ years of IT experience focusing on enterprise data architecture and management.
• Must be able to obtain a Public Trust security clearance.
• Advanced understanding of ETL and ELT processes.

Responsibilities

• Create, maintain, and manage Spark Structured Streaming jobs.
• Plan, create, and maintain data architectures.
• Support AI/ML teams with optimizing feature engineering code.

Skills

ETL and ELT tools

Spark

SQL

Data Lake concepts

Python

Data quality frameworks

Education

Bachelor's degree in Computer Science or related discipline

Tools

Databricks

Docker

AWS

Kafka

Jenkins

Job description

Pyramid Systems, Inc. is seeking a Senior Data Engineer to create innovative architecture solutions. Applicants should have at least 8 years of IT experience with enterprise data architecture and management, including expertise in Databricks, ETL/ELT tools, and advanced SQL. Responsibilities include managing data pipelines, supporting AI/ML teams, and ensuring data quality. The position offers a competitive salary of USD $106,131 - $159,196, along with benefits such as an Employee Stock Ownership Program and a flexible workplace approach.