Databricks Architect -- KATDC5738914

Compunnel Inc.; Edison (NJ); On-site; USD 100,000 - 130,000; Full time

Job summary

A leading technology company located in Edison, New Jersey is seeking a Data Engineer to design, develop, and maintain data ingestion pipelines using Databricks and Lakeflow Connect. The ideal candidate should have at least 4 years of experience in data engineering, hands-on experience with Databricks SQL and PySpark, and a strong understanding of data integration patterns. This position requires collaboration with data architects and business stakeholders to ensure data quality and compliance, along with optimizing ingestion performance for scalability and cost efficiency.

Qualifications

• 4 years of experience in data engineering or ETL development.
• Hands-on experience with Databricks SQL, PySpark, and Delta Lake.
• Proficiency with Lakeflow Connect for building data ingestion workflows.

Responsibilities

• Design, develop, and maintain data ingestion pipelines using Databricks.
• Integrate data from various sources into Delta Lake.
• Optimize data ingestion performance ensuring scalability and reliability.

Skills

Databricks SQL

PySpark

Delta Lake

Data Ingestion Workflows

Cloud Platforms (Azure, AWS, GCP)

Data Modeling

Collaboration Skills

CICD Practices

Education

Bachelor's or Master's degree in Computer Science, Information Systems, Data Engineering

Tools

Lakeflow Connect

Git

Job description

Design develop and maintain data ingestion pipelines using Databricks and Lakeflow Connect

Integrate data from various structured and unstructured sources into Delta Lake and other data storage systems

Implement realtime and batch ingestion workflows to support analytics and reporting needs

Optimize data ingestion performance ensuring scalability reliability and cost efficiency

Collaborate with data architects analysts and business stakeholders to define data requirements and ingestion strategies

Ensure data quality lineage and governance compliance across the ingestion process

Stay up to date with emerging Databricks Lakehouse and data integration technologies and best practices

Required Qualifications

• Bachelors or Masters degree in Computer Science Information Systems Data Engineering or a related field
• 4 years of experience in data engineering or ETL development
• Handson experience with Databricks SQL PySpark Delta Lake
• Proficiency with Lakeflow Connect for building and managing data ingestion workflows
• Strong understanding of data integration patterns data modeling and data lakehouse architectures
• Experience with cloud platforms Azure AWS or GCP and associated data services
• Knowledge of CICD version control Git and infrastructure as code practices
• Familiarity with data governance security and compliance standards

Preferred Skills

• Experience with streaming technologies Kafka Event Hubs etc
• Knowledge of REST APIs and connectorbased ingestion
• Exposure to machine learning data pipelines in Databricks
• Strong problemsolving communication and collaboration skills