Azure Databricks Data Engineer – Spark & Lakehouse

Signature IT World Inc; Pennsylvania; Hybrid; USD 120.000 - 150.000; Vollzeit

Zusammenfassung

Signature IT World Inc. is seeking an experienced Data Engineer based in Princeton, NJ, with hybrid work options. The role focuses on building scalable data pipelines using Databricks, Spark, PySpark, and SQL to support ETL/ELT processes.

You will work with Delta Lake and Lakehouse architectures, optimize Spark workloads, and ensure data quality while coordinating with analysts, architects, and business teams across cloud platforms such as Azure and AWS.

Qualifikationen

• Strong expertise in Databricks platform and Spark-based pipelines.
• Proficient PySpark and Spark SQL skills.
• Experience building robust ETL/ELT pipelines and data workflows.

Aufgaben

• Design, develop, and maintain scalable data pipelines using Databricks and Apache Spark.
• Develop complex data transformations using PySpark and Spark SQL.
• Build batch and streaming data processing solutions.
• Implement ETL/ELT workflows to ingest data from multiple sources.
• Work with Delta Lake, Delta Tables, and Lakehouse architecture.
• Optimize Spark jobs for performance, scalability, and cost efficiency.
• Implement partitioning, caching, broadcasting, and other Spark optimization techniques.
• Develop and maintain data workflows using Databricks Workflows and Jobs.
• Implement incremental data loading, CDC, and SCD Type 1/Type 2.
• Ensure data quality, data validation, and pipeline reliability.
• Work with cloud storage and data services on Azure/AWS.
• Implement security, access control, and governance using Unity Catalog.
• Collaborate with data analysts, architects, and business teams.

Kenntnisse

Databricks

Apache Spark

PySpark

SQL

Tools

Delta Lake

Delta Tables

Lakehouse

Jobbeschreibung

Signature IT World Inc. is seeking an experienced Data Engineer based in Princeton, NJ, with hybrid work options. The role focuses on building scalable data pipelines using Databricks, Spark, PySpark, and SQL to support ETL/ELT processes.

You will work with Delta Lake and Lakehouse architectures, optimize Spark workloads, and ensure data quality while coordinating with analysts, architects, and business teams across cloud platforms such as Azure and AWS.