Sr. Data Engineer

Inizio Partners Corp; Jersey City (NJ); On-site; USD 100,000 - 130,000; Full time

Job summary

A leading technology consulting firm seeks an experienced Data Engineer to design and implement data pipelines using AWS and Kafka. The ideal candidate will have strong skills in Python, SQL, and data engineering processes. You will work with multiple stakeholders to ensure data accuracy and reliability, contributing to informed business decisions. This full-time role is based in Jersey City, NJ, and offers competitive pay and benefits.

Qualifications

• 3+ years of AWS Console, S3, Lambda, Kinesis, Glue, and CloudWatch.
• 3+ years of Kafka streams experience.
• 5+ years of data engineering, data pipeline development, and ETL experience using Python, SQL, and Snowflake.
• Experience with REST and SOAP APIs data transformation.
• Strong understanding of dimensional modeling principles.

Responsibilities

• Implement and manage AWS tools and environment.
• Build and maintain data connections to various sources feeding into a Snowflake database.
• Ensure data consistency and reliability as business requirements change.

Skills

AWS Console

Kafka streams

Python

SQL

ETL processes

Data analysis

Data visualization

Container orchestration

Education

Bachelors degree in information systems, computer science, or related technical field

Tools

Snowflake

AWS S3

Apache Kafka

Tableau

Docker

Job description

Our client is expanding their Data Engineering toolset and need an experienced AWS, Kafka streams, Python and SQL Data Engineer. This role will be required to design and implement our new data pipelines using AWS and Kafka streams, and train other team members. As a Sr. Data Engineer on our Data Engineering team, you will also design, write, scale, and maintain complex data pipelines using Python within our development framework. You will contribute to the organization's success by partnering with business, analytics, and infrastructure teams to design and build datasets to facilitate the business' decisions and increase effectiveness. Collaborating across disciplines, you will identify internal and external data sources to design pipelines, table structures, define ETL strategies and automate error-handling and validation. This team works with various stakeholders and divisions, including the executive team, with the goal of providing timely, accurate, and reliable data to thousands of users. Your role will be critical in defining the appropriate architecture and processes to support our Snowflake data infrastructure, that is flexible, agile, reliable, responsive, and scalable. You will be a member of a small, high-performing team with responsibilities to support the data warehouse and analytics teams throughout the company. As a member of the Data Engineering team, you will report to the Manager of Data Engineering.

Primary Responsibilities

• Implement and manage AWS tools and environment.
• Implement and manage Apache Kafka streams.
• Using Python, build and maintain multiple source data connections to API's, SQL databases, AWS S3, and other sources feeding into a Snowflake Clouds database system.
• Work with internal and external users and providers to build datasets that add value to the business and allow for informed business decisions.
• Ensure data consistency, accuracy and reliability as data and business requirements change.

Required Skills

• 3+ years of AWS Console, S3, Lambda, Kinesis, Glue, and CloudWatch.
• 3+ years of Kafka streams experience.
• 5+ years of data engineering, data pipeline development, and ETL experience using Python, SQL, and Snowflake
• Experience requesting, transforming, and ingesting data from REST and SOAP APIs
• Proficiency in the Python scripting language, SQL, Cloud databases, and ETL development processes & tools
• Strong understanding of traditional relational databases, data and dimensional modeling principles, and data normalization techniques
• Ability to initiate, drive, and manage projects with competing priorities.
• Ability to communicate effectively with business leaders, IT leadership, and engineers.
• Must have a passion for data and helping the business turn data into information and action.

Bonus Skills

• Experience with data streaming technologies like PySpark
• Experience with pipeline technologies like DBT, Apache Airflow or FiveTran
• Experience with MPP technologies and databases
• Experience with data visualization tools like Tableau or Sigma
• Experience with container orchestration tools like Docker or Kubernetes
• Experience with Azure data product offerings and platform
• Experience working with Salesforce and SAP data.
• Experience using Terraform or other infrastructure as code tools.

Required Education/Experience

• Bachelors degree in information systems, computer science, or related technical field