Senior Databricks Data Engineer — Python & Spark

Veriipro; Jersey City (NJ); On-site; USD 120,000 - 190,000; Full time

Job summary

Veriipro in Jersey City, NJ is seeking an senior Data Engineer to design and maintain scalable data pipelines using Databricks, Python, and PySpark. You will collaborate with architects and analysts to deliver robust data solutions across multiple sources.

The role emphasizes data quality, governance, and performance tuning, with opportunities to optimize batch and real-time workflows in a fast-paced environment.

Qualifications

• 10+ years in Data Engineering, Data Warehousing, or Big Data development.
• 5+ years with Databricks, Python, and PySpark.
• Experience building large-scale ETL/ELT data pipelines.
• Strong knowledge of Spark SQL, Delta Lake, and Data Lakehouse architecture.
• Experience with structured and unstructured data from multiple sources.
• Strong SQL skills with relational databases.
• Data modeling, transformation, and data quality processes.
• Knowledge of orchestration tools like Airflow or ADF.
• Experience with Git, CI/CD, and Agile methodologies.
• Strong debugging, performance tuning, and communication skills.

Responsibilities

• Design, develop, and maintain data pipelines using Databricks, Python, and PySpark.
• Build scalable ingestion, transformation, and integration solutions.
• Collaborate with Data Architects, Analysts, and business teams to deliver data solutions.
• Develop and optimize batch and real-time data processing workflows.
• Ensure data quality, integrity, security, and governance standards.
• Monitor, troubleshoot, and resolve production data pipeline issues.
• Optimize Spark jobs, SQL queries, and Databricks workloads for performance.
• Participate in data modeling, code reviews, and platform enhancements.
• Create technical documentation and support production deployments.
• Participate in Agile ceremonies and drive continuous improvement initiatives.

Skills

Analytical thinking

Data analysis

Communication

Collaboration

Problem solving

Tools

Databricks

Python

PySpark

Spark SQL

Delta Lake

Airflow

ADF

Git

CI/CD

Job description

Veriipro in Jersey City, NJ is seeking an senior Data Engineer to design and maintain scalable data pipelines using Databricks, Python, and PySpark. You will collaborate with architects and analysts to deliver robust data solutions across multiple sources.

The role emphasizes data quality, governance, and performance tuning, with opportunities to optimize batch and real-time workflows in a fast-paced environment.