Lead Data Engineer (Hybrid NJ) — Cloud, Spark & Delta Lake

Appiness Inc.; Town of Newark (WI); Hybrid; USD 140,000 - 200,000; Full time

Job summary

Appiness Inc. is seeking a Senior Data Engineer / Lead Data Engineer to design and maintain scalable ETL/ELT pipelines for large datasets.

You will leverage Python, PySpark, Spark, and SQL across Azure and AWS, with Databricks, Delta Lake, and Snowflake to deliver robust data solutions. You will orchestrate pipelines with Airflow and related tools, implement monitoring, and collaborate on CI/CD workflows using GitHub, Jenkins, Docker, and Azure DevOps.

Qualifications

• 8 years of experience in Data Engineering or related roles.
• Strong hands-on experience with Python, PySpark, Apache Spark, and SQL.
• Strong experience with Azure and/or AWS cloud environments.
• Experience with Databricks and Delta Lake.
• Strong knowledge of Azure Data Factory, Azure Synapse, Azure Data Lake/Blob Storage.
• Experience with AWS services such as S3, Glue, Athena, DMS, Lambda, SNS, SQS, and EventBridge.
• Strong experience with Snowflake and cloud data warehousing.
• Strong understanding of ETL/ELT, data warehousing, data modeling, and data integration.
• Experience working with Oracle, SQL Server, MySQL, Hive, and other relational databases.
• Strong programming and scripting experience using Python, SQL, and Shell scripting.
• Experience with Git, Bitbucket/GitHub, Jenkins, Docker, and Azure DevOps.

Responsibilities

• Design, develop, and maintain scalable ETL/ELT data pipelines for large-volume structured and unstructured data.
• Develop data processing solutions using Python, PySpark, Apache Spark, and SQL.
• Build and optimize data pipelines using Databricks, Azure Data Factory, AWS Glue, and Snowflake.
• Work with both Azure and AWS cloud platforms to implement modern data engineering solutions.
• Work with Delta Lake, Azure Data Lake, Amazon S3, Azure Blob Storage, Azure Synapse, and Snowflake.
• Develop pipeline orchestration and scheduling using Airflow, Control-M, Databricks Workflows, Azure Data Factory, and AWS services.
• Implement monitoring, logging, alerting, and troubleshooting processes for production data pipelines.
• Work with CI/CD processes using GitHub, Bitbucket, Jenkins, Docker, and Azure DevOps.

Skills

ETL/ELT pipelines

Python

PySpark

Apache Spark

SQL

Azure

AWS

Databricks

Delta Lake

Azure Data Factory

Azure Synapse

Azure Data Lake

Blob Storage

Snowflake

Airflow

Databricks Workflows

GitHub

Bitbucket

Jenkins

Docker

Azure DevOps

S3

Glue

Athena

DMS

Lambda

SNS

SQS

EventBridge

Oracle

SQL Server

MySQL

Hive

Shell scripting

Tools

Databricks

Delta Lake

Azure Data Factory

Snowflake

Airflow

Control-M

Databricks Workflows

AWS Glue

Azure DevOps

GitHub

Bitbucket

Jenkins

Docker

Job description

Appiness Inc. is seeking a Senior Data Engineer / Lead Data Engineer to design and maintain scalable ETL/ELT pipelines for large datasets.

You will leverage Python, PySpark, Spark, and SQL across Azure and AWS, with Databricks, Delta Lake, and Snowflake to deliver robust data solutions. You will orchestrate pipelines with Airflow and related tools, implement monitoring, and collaborate on CI/CD workflows using GitHub, Jenkins, Docker, and Azure DevOps.