Senior Data Engineer: Databricks Lakehouse & Data Pipelines

Bitful Consulting; Pennsylvania; On-site; USD 120,000 - 180,000; Full time

Job summary

Bitful Consulting in Pennsylvania is seeking an experienced Senior Data Engineer to design and maintain scalable data pipelines in a Databricks lakehouse environment. You will ingest, clean and transform data from multiple sources using Python, PySpark and SQL, and build models and datasets for reporting in Tableau and Spotfire.

You will collaborate with analysts, architects and business stakeholders to deliver practical data solutions, improve pipeline reliability, and mentor other engineers.

Qualifications

• 7+ years of experience in data engineering and production pipelines.
• Strong SQL and data warehousing knowledge.
• Hands-on Databricks, Apache Spark, and cloud data platforms.
• Proficient in Python and PySpark.
• Experience in ETL/ELT pipelines across multiple sources.
• Experience preparing datasets for Tableau or BI tools.
• Familiarity with source control, automated testing and CI/CD.
• Ability to troubleshoot data issues and optimize pipelines.
• Strong communication and collaboration with technical teams.
• Experience with Spotfire, Oracle, Informatica, or R preferred.
• Databricks certification preferred.

Responsibilities

• Develop and maintain scalable data pipelines and reusable datasets within a Databricks lakehouse environment.
• Ingest, validate, cleaning and transforming structured and unstructured data from multiple sources using Python, PySpark and SQL.
• Build data models and reporting datasets that support Tableau and Spotfire.
• Integrate data warehouse and reporting systems with modern cloud analytics platforms.
• Improve pipeline reliability, data quality and processing performance.
• Implement automated testing, monitoring, documentation and CI/CD practices.
• Partner with analysts, architects, developers and business stakeholders to deliver practical data solutions.
• Provide technical guidance, reviewing code and mentoring other data engineer

Skills

Data engineering

SQL

Python

PySpark

ETL/ELT development

Data modeling

Data warehousing

Communication

Troubleshooting

Mentoring

Tools

Databricks

Apache Spark

Tableau

Spotfire

Oracle

Informatica

R

Job description

Bitful Consulting in Pennsylvania is seeking an experienced Senior Data Engineer to design and maintain scalable data pipelines in a Databricks lakehouse environment. You will ingest, clean and transform data from multiple sources using Python, PySpark and SQL, and build models and datasets for reporting in Tableau and Spotfire.

You will collaborate with analysts, architects and business stakeholders to deliver practical data solutions, improve pipeline reliability, and mentor other engineers.