Senior Data Engineer: Remote Databricks & PySpark Expert

Propelis; United States; Remote; USD 140,000 - 190,000; Full time

Job summary

Propelis is seeking an experienced data engineer to design, develop, and maintain scalable data pipelines using Databricks and PySpark. You will work with large-scale structured and unstructured data from diverse sources and collaborate with data scientists, analysts, and business stakeholders to understand data needs.

You will optimize data workflows for performance and reliability, implement data quality checks, monitoring, and alerting, and ensure data security and governance compliance while

Qualifications

• 8-12 years of experience in data engineering or related roles.
• Strong hands-on experience with Databricks and Apache Spark (PySpark).
• Proficiency in Python and SQL.
• Experience with cloud platforms such as Azure, AWS, or GCP (preferably Azure).
• Solid understanding of ETL/ELT processes, data warehousing, and data modeling.
• Familiarity with Delta Lake and Parquet formats.
• Experience with CI/CD pipelines and version control (e.g., Git).
• Excellent problem-solving and communication skills.

Responsibilities

• Design, develop, and maintain scalable data pipelines using Databricks and PySpark.
• Work with large-scale structured and unstructured data from various sources.
• Optimize data workflows for performance and reliability.
• Collaborate with data scientists, analysts, and business stakeholders to understand data requirements.
• Implement data quality checks, monitoring, and alerting mechanisms.
• Ensure data security and governance policy compliance.
• Participate in code reviews and contribute to best practices in data engineering.

Skills

Python

SQL

Data modeling

Cloud platforms

Data warehousing

Delta Lake

Parquet

ETL/ELT

Communication

Problem solving

Tools

Databricks

Apache Spark (PySpark)

Git

CI/CD

Job description

Propelis is seeking an experienced data engineer to design, develop, and maintain scalable data pipelines using Databricks and PySpark. You will work with large-scale structured and unstructured data from diverse sources and collaborate with data scientists, analysts, and business stakeholders to understand data needs.

You will optimize data workflows for performance and reliability, implement data quality checks, monitoring, and alerting, and ensure data security and governance compliance while