Databricks Lakehouse Data Ingestion Architect

Compunnel Inc.; Edison (NJ); On-site; USD 100,000 - 130,000; Full time

Job summary

A leading technology company located in Edison, New Jersey is seeking a Data Engineer to design, develop, and maintain data ingestion pipelines using Databricks and Lakeflow Connect. The ideal candidate should have at least 4 years of experience in data engineering, hands-on experience with Databricks SQL and PySpark, and a strong understanding of data integration patterns. This position requires collaboration with data architects and business stakeholders to ensure data quality and compliance, along with optimizing ingestion performance for scalability and cost efficiency.

Qualifications

• 4 years of experience in data engineering or ETL development.
• Hands-on experience with Databricks SQL, PySpark, and Delta Lake.
• Proficiency with Lakeflow Connect for building data ingestion workflows.

Responsibilities

• Design, develop, and maintain data ingestion pipelines using Databricks.
• Integrate data from various sources into Delta Lake.
• Optimize data ingestion performance ensuring scalability and reliability.

Skills

Databricks SQL

PySpark

Delta Lake

Data Ingestion Workflows

Cloud Platforms (Azure, AWS, GCP)

Data Modeling

Collaboration Skills

CICD Practices

Education

Bachelor's or Master's degree in Computer Science, Information Systems, Data Engineering

Tools

Lakeflow Connect

Git

Job description

A leading technology company located in Edison, New Jersey is seeking a Data Engineer to design, develop, and maintain data ingestion pipelines using Databricks and Lakeflow Connect. The ideal candidate should have at least 4 years of experience in data engineering, hands-on experience with Databricks SQL and PySpark, and a strong understanding of data integration patterns. This position requires collaboration with data architects and business stakeholders to ensure data quality and compliance, along with optimizing ingestion performance for scalability and cost efficiency.