Senior Data Engineer

Sparibis; United States; On-site; USD 100,000 - 720,000; Full time

Job summary

An established industry player is seeking a seasoned Data Engineer with over a decade of experience in enterprise data architecture and management. In this pivotal role, you will design and implement robust data architectures and pipelines, ensuring alignment with business objectives. Your expertise in tools like Databricks and advanced SQL will be crucial in optimizing data processes and supporting AI/ML initiatives. Join a dynamic team that values innovation and collaboration, where your contributions will drive impactful solutions in the realm of data management. If you are passionate about leveraging data to inform decision-making, this opportunity is perfect for you.

Benefits

Medical Insurance

Vision Insurance

401(k)

Disability Insurance

Qualifications

• 10+ years of IT experience focusing on enterprise data architecture and management.
• Experience with Databricks and ETL tools like SSIS and Pentaho.
• Advanced SQL skills including joins and performance optimization.

Responsibilities

• Plan and maintain data architectures aligned with business needs.
• Create and manage data pipelines and transformations.
• Optimize data processes and automate manual tasks.

Skills

Enterprise Data Architecture

Databricks

Structured Streaming

Delta Lake

ETL Tools (SSIS, Pentaho)

SQL (Advanced)

Spark

Python

Data Lake Concepts

Data Quality Frameworks

Education

Bachelor's Degree in IT

Tools

AWS

Docker

Jenkins

Kafka

ksqlDB

Job description

Years’ Experience: 10+ years professional experience

Education: Bachelor degree in IT related field

Clearance: Applicants must be able to obtain and maintain up to a Public Trust clearance. United States Citizenship is required as part of the eligibility criteria to be able to obtain this type of security clearance.

Key Skills:

• 10+ years of IT experience focusing on enterprise data architecture and management.
• Experience with Databricks, Structured Streaming, Delta Lake concepts, and Delta Live Tables required.
• Experience with ETL and ELT tools such as SSIS, Pentaho, and/or Data Migration Services.
• Advanced level SQL experience (Joins, Aggregation, Windowing functions, Common Table Expressions, RDBMS schema design, Postgres performance optimization).

Responsibilities

• Plan, create, and maintain data architectures, ensuring alignment with business requirements.
• Obtain data, formulate dataset processes, and store optimized data.
• Identify problems and inefficiencies and apply solutions.
• Determine tasks where manual participation can be eliminated with automation.
• Identify and optimize data bottlenecks, leveraging automation where possible.
• Create and manage data lifecycle policies (retention, backups/restore, etc).
• In-depth knowledge for creating, maintaining, and managing ETL/ELT pipelines.
• Create, maintain, and manage data transformations.
• Create, maintain, and manage data pipeline schedules.
• Create, maintain, and manage data quality gates (Great Expectations) to ensure high data quality.
• Support AI/ML teams with optimizing feature engineering code.
• Expertise in Spark/Python/Databricks, Data Lake and SQL.
• Create, maintain, and manage Spark Structured Steaming jobs, including using the newer Delta Live Tables and/or DBT.
• Research existing data in the data lake to determine best sources for data.
• Create, manage, and maintain ksqlDB and Kafka Streams queries/code.
• Maintain and update Python-based data processing scripts executed on AWS Lambdas.
• Unit tests for all the Spark, Python data processing and Lambda codes.
• Maintain PCIS Reporting Database data lake with optimizations and maintenance (performance tuning, etc).
• Streamlining data processing experience including formalizing concepts of how to handle lake data, defining windows, and how window definitions impact data freshness.
• Perform related duties as assigned.

Qualifications

• 10+ years of IT experience focusing on enterprise data architecture and management.
• Must be able to obtain a Public Trust security clearance.
• Experience in Conceptual/Logical/Physical Data Modeling & expertise in Relational and Dimensional Data Modeling.
• Experience with Databricks, Structured Streaming, Delta Lake concepts, and Delta Live Tables required.
• Additional experience with Spark, Spark SQL, Spark DataFrames and DataSets, and PySpark.
• Data Lake concepts such as time travel and schema evolution and optimization
• Structured Streaming and Delta Live Tables with Databricks a bonus.
• Experience leading and architecting enterprise-wide initiatives specifically system integration, data migration, transformation, data warehouse build, data mart build, and data lakes implementation / support.
• Advanced level understanding of streaming data pipelines and how they differ from batch systems.
• Formalize concepts of how to handle late data, defining windows, and data freshness.
• Advanced understanding of ETL and ELT and ETL/ELT tools such as SSIS, Pentaho, Data Migration Service etc.
• Understanding of concepts and implementation strategies for different incremental data loads such as tumbling window, sliding window, high watermark, etc.
• Familiarity and/or expertise with Great Expectations or other data quality/data validation frameworks a bonus.
• Understanding of streaming data pipelines and batch systems.
• Familiarity with concepts such as late data, defining windows, and how window definitions impact data freshness.
• Advanced level SQL experience (Joins, Aggregation, Windowing functions, Common Table Expressions, RDBMS schema design, Postgres performance optimization)
• Indexing and partitioning strategy experience.
• Debug, troubleshoot, design and implement solutions to complex technical issues.
• Experience with large-scale, high-performance enterprise big data application. deployment and solution.
• Understanding how to create DAGs to define workflows.
• Familiarity with CI/CD pipelines, containerization, and pipeline orchestration tools such as Airflow, Prefect, etc a bonus but not required.
• Architecture experience in AWS environment a bonus.
• Familiarity working with Kinesis and/or Lambda specifically with how to push and pull data, how to use AWS tools to view data in Kinesis streams, and for processing massive data at scale a bonus.
• Experience with Docker, Jenkins, and CloudWatch
• Ability to write and maintain Jenkinsfiles for supporting CI/CD pipelines.
• Experience working with AWS Lambdas for configuration and optimization.
• Experience working with DynamoDB to query and write data.
• Experience with S3.
• Knowledge of Python (Python 3 desired) for CI/CD pipelines a bonus.
• Familiarity with Pytest and Unittest a bonus.
• Experience working with JSON and defining JSON Schemas a bonus.
• Experience setting up and management Confluent/Kafka topics and ensuring performance using Kafka a bonus.
• Familiarity with Schema Registry, message formats such as Avro, ORC, etc.
• Understanding how to manage ksqlDB SQL files and migrations and Kafka Streams.
• Ability to thrive in a team-based environment.
• Experience briefing the benefits and constraints of technology solutions to technology partners, stakeholders, team members, and senior level of management.

Seniority level

• Seniority level

  Mid-Senior level

Employment type

• Employment type

  Full-time

Job function

• Job function

  Information Technology

• Industries

  IT Services and IT Consulting

Referrals increase your chances of interviewing at Sparibis by 2x

Inferred from the description for this job

Medical insurance

Vision insurance

401(k)

Disability insurance

Get notified about new Data Engineer jobs in United States.

United States $170,000.00-$720,000.00 1 week ago

Washington DC-Baltimore Area $100,000.00-$130,000.00 4 days ago

Data Visualization Engineer (L5) - Product

United States $170,000.00-$720,000.00 1 week ago

United States $170,000.00-$720,000.00 9 hours ago

New York, NY $126,100.00-$186,800.00 1 week ago

Seattle, WA $126,100.00-$186,800.00 1 week ago

Austin, TX $126,100.00-$186,800.00 1 week ago

United States $126,100.00-$186,800.00 1 week ago

Washington DC-Baltimore Area $120,000.00-$135,000.00 3 weeks ago

Mountain View, CA $126,100.00-$186,800.00 1 week ago

United States $114,000.00-$171,000.00 2 weeks ago

United States $100,000.00-$720,000.00 1 week ago

Data Engineer, Analytics (Technical Leadership)

United States $206,000.00-$281,000.00 2 weeks ago

United States $173,000.00-$242,000.00 2 weeks ago

United States $145,000.00-$204,000.00 2 weeks ago

Washington DC-Baltimore Area $110,000.00-$120,000.00 3 days ago

United States $100,000.00-$720,000.00 1 week ago

We’re unlocking community knowledge in a new way. Experts add insights directly into each article, started with the help of AI.