Azure Engineer

Signature IT World Inc; Pennsylvania; Hybrid; USD 120,000 - 150,000; Full time

Job summary

Signature IT World Inc. is seeking an experienced Data Engineer based in Princeton, NJ, with hybrid work options. The role focuses on building scalable data pipelines using Databricks, Spark, PySpark, and SQL to support ETL/ELT processes.

You will work with Delta Lake and Lakehouse architectures, optimize Spark workloads, and ensure data quality while coordinating with analysts, architects, and business teams across cloud platforms such as Azure and AWS.

Qualifications

• Strong expertise in Databricks platform and Spark-based pipelines.
• Proficient PySpark and Spark SQL skills.
• Experience building robust ETL/ELT pipelines and data workflows.

Responsibilities

• Design, develop, and maintain scalable data pipelines using Databricks and Apache Spark.
• Develop complex data transformations using PySpark and Spark SQL.
• Build batch and streaming data processing solutions.
• Implement ETL/ELT workflows to ingest data from multiple sources.
• Work with Delta Lake, Delta Tables, and Lakehouse architecture.
• Optimize Spark jobs for performance, scalability, and cost efficiency.
• Implement partitioning, caching, broadcasting, and other Spark optimization techniques.
• Develop and maintain data workflows using Databricks Workflows and Jobs.
• Implement incremental data loading, CDC, and SCD Type 1/Type 2.
• Ensure data quality, data validation, and pipeline reliability.
• Work with cloud storage and data services on Azure/AWS.
• Implement security, access control, and governance using Unity Catalog.
• Collaborate with data analysts, architects, and business teams.

Skills

Databricks

Apache Spark

PySpark

SQL

Tools

Delta Lake

Delta Tables

Lakehouse

Job description

Location: Princeton NJ-Hybrid

Contract Duration: 12months

We are looking for an experienced Data Engineer with strong expertise in Databricks, Apache Spark, PySpark, and SQL to design, develop, and maintain scalable data engineering solutions. The candidate will be responsible for building robust ETL/ELT pipelines, processing large volumes of structured and unstructured data, and implementing efficient data transformation workflows using the Databricks platform.

Key Responsibilities

• Design, develop, and maintain scalable data pipelines using Databricks and Apache Spark.
• Develop complex data transformations using PySpark and Spark SQL.
• Build batch and streaming data processing solutions.
• Implement ETL/ELT workflows to ingest data from multiple sources.
• Work with Delta Lake, Delta Tables, and Lakehouse architecture.
• Optimize Spark jobs for performance, scalability, and cost efficiency.
• Implement partitioning, caching, broadcasting, and other Spark optimization techniques.
• Develop and maintain data workflows using Databricks Workflows and Jobs.
• Implement incremental data loading, CDC, and SCD Type 1/Type 2.
• Ensure data quality, data validation, and pipeline reliability.
• Work with cloud storage and data services on Azure/AWS.
• Implement security, access control, and governance using Unity Catalog.
• Collaborate with data analysts, architects, and business teams.