Data Engineer - Databricks & Python Data Pipelines

Cloud Analytics Technologies, LLC; Jersey City (NJ); On-site; USD 83,000 - 124,000; Full time

Job summary

Cloud Analytics Technologies, LLC is seeking an experienced Data Engineer to design, build, and optimize scalable data pipelines in Jersey City, NJ. You will leverage Databricks, Python, and SQL to support analytics, BI, and ML workloads across the organization.

The role emphasizes ETL/ELT development, data modeling, and implementing data quality, governance, and security measures while collaborating with Data Scientists and Cloud Engineers.

Qualifications

• Strong SQL and query tuning experience.
• Experience with large-scale data processing.
• Proficiency in Python for data pipelines and ETL/ELT development.

Responsibilities

• Design, build, and maintain ETL/ELT workflows on Databricks.
• Develop scalable data pipelines using Python and Spark.
• Optimize data transformation logic for performance and cost efficiency.
• Work with structured and unstructured data from various sources.
• Develop and manage data models, schemas, and tables across databases (SQL/NoSQL).
• Implement data quality, validation, and monitoring frameworks.
• Collaborate with Data Scientists, Analysts, and Cloud Engineers to support analytics and ML workloads.
• Ensure data security, governance, and compliance in all solutions.
• Troubleshoot pipeline failures and continuously improve pipeline reliability.
• Integrate data from APIs, streaming sources, and cloud storage.

Skills

SQL proficiency

Python

Data modeling

Data governance

Cloud platforms

Tools

Databricks

ADF

Glue

Dataflow

Job description

Cloud Analytics Technologies, LLC is seeking an experienced Data Engineer to design, build, and optimize scalable data pipelines in Jersey City, NJ. You will leverage Databricks, Python, and SQL to support analytics, BI, and ML workloads across the organization.

The role emphasizes ETL/ELT development, data modeling, and implementing data quality, governance, and security measures while collaborating with Data Scientists and Cloud Engineers.