Data Engineer - Databricks Pipelines & Data Quality

Artifintech

Jersey City (NJ)

On-site

USD 55,000 - 96,000

Full time

14 hours ago

Be an early applicant

A complete application in a minute

Tailored resume and cover letter, ready to send.

Get past ATS filters

Job summary

ARTIFICIAL INTELLIGENCE TECHNOLOGIES LLC in Jersey City, NJ is seeking an experienced Data Engineer to design, build, and optimize scalable data pipelines using Databricks, Python, and Spark. You will work across structured and unstructured data sources, ensuring data quality and governance while supporting analytics and ML workloads.

The role requires strong SQL, ETL/ELT expertise, and experience with cloud data platforms (AWS/Azure/GCP).

Qualifications

Experience designing and maintaining ETL/ELT pipelines on Databricks.
Strong Python and Spark programming skills.
Extensive SQL, query tuning, and schema design expertise.
Experience with large-scale datasets and NoSQL/SQL databases.
Ability to implement data quality, validation, and monitoring frameworks.
Experience with cloud platforms (AWS/Azure/GCP) and data integration tools.

Responsibilities

Design, build, and maintain ETL/ELT workflows on Databricks.
Develop scalable data pipelines using Python and Spark.
Optimize data transformation logic for performance and cost efficiency.
Work with structured and unstructured data from various sources.
Develop and manage data models, schemas, and tables across databases (SQL/NoSQL).
Implement data quality, validation, and monitoring frameworks.
Collaborate with Data Scientists, Analysts, and Cloud Engineers to support analytics and ML workloads.
Ensure data security, governance, and compliance in all solutions.
Troubleshoot pipeline failures and continuously improve pipeline reliability.
Integrate data from APIs, streaming sources, and cloud storage.