Senior Databricks Python Engineer (Data Lakehouse, PySpark)

Envision Technology Solutions; New York (NY); Hybrid; USD 120,000 - 180,000; Full time

Job summary

Envision Technology Solutions in New York City, NY / Jersey City, NJ is seeking a senior Data Engineer to design and optimize large-scale data pipelines and data lakehouse architectures. You will work with Databricks, Python, PySpark, Spark SQL and Delta Lake to enable robust analytics across diverse data sources.

The role requires strong SQL, data modelling, and data quality skills, plus experience with orchestration tools like Airflow and CI/CD in an Agile environment; excellent communication

Qualifications

• 10+ years of experience in data engineering, data warehousing or big data development.
• 5+ years hands-on experience with Databricks, Python and PySpark.
• Strong hands-on Databricks, Python, PySpark development.
• Experience building large-scale ETL/ELT data pipelines.
• Strong knowledge of Spark SQL, Delta Lake and Data Lakehouse.
• Experience with structured and unstructured data from multiple sources.
• Strong SQL skills and experience with relational databases.
• Experience with data modelling, data transformation and data quality.
• Knowledge of orchestration tools such as Airflow, ADF, or similar.
• Experience with Git, CI/CD and Agile development.
• Strong analytical, debugging and performance tuning skills.
• Excellent verbal and written communication skills.

Skills

Data engineering

Data warehousing

Big Data development

Databricks

Python

PySpark

ETL/ELT pipelines

Spark SQL

Delta Lake

Data Lakehouse

SQL

Data modelling

Data transformation

Data quality

Airflow

ADF

Git

CI/CD

Agile

Analytical skills

Debugging

Communication skills

Tools

Airflow

ADF

Git

CI/CD

Job description

Envision Technology Solutions in New York City, NY / Jersey City, NJ is seeking a senior Data Engineer to design and optimize large-scale data pipelines and data lakehouse architectures. You will work with Databricks, Python, PySpark, Spark SQL and Delta Lake to enable robust analytics across diverse data sources.

The role requires strong SQL, data modelling, and data quality skills, plus experience with orchestration tools like Airflow and CI/CD in an Agile environment; excellent communication