Senior Data Engineer: Python PySpark Databricks (Remote)

AB2 Consulting; United States; Remote; USD 140,000 - 190,000; Full time

Job summary

AB2 Consulting is seeking a Senior Data Engineer to join a remote team. The role centers on building and optimizing data‑driven backend systems using Python, PySpark, and Databricks, with a focus on scalable data processing and API integration.

You will collaborate with architects, analysts, and ops, review code, mentor juniors, and drive performance improvements across data platforms while maintaining high quality and reliability.

Qualifications

• 7+ years IT development experience is required.
• Hands-on data‑driven backend development experience.
• Strong skills in Python/PySpark, Databricks, SQL, Java or C#, and API integration.

Responsibilities

• Understand system architecture and support operations with monitoring and dashboards.
• Collaborate with architects and technical leads on design and implementation.
• Monitor, optimize performance, scalability, and costs of data pipelines and services.
• Work with Business Analysts to implement integration requirements.
• Troubleshoot performance, backend, data, and functional issues.
• Review code and mentor junior engineers; participate in cross‑functional delivery.

Skills

Python/PySpark

Databricks

SQL/PLSQL

Java/C#

Backend/API integration

Code reviews

Mentoring

Job description

Job Title : Senior Data Engineer Python/PySpark & Databricks

Work Location: Remote

Experience : 8Years

Job Summary: We are looking for an experienced Backend/Data Developer with hands-on experience in Databricks, PySpark, Python/Java, Web Services, and SQL to support Dealer Services (Data Edge). The ideal candidate will have strong backend and data engineering skills, experience designing and troubleshooting data-driven solutions, and the ability to work closely with architects, technical leads, business analysts, and operations teams.

Essential Job Functions

• Understand the overall system architecture and support operations with monitoring, dashboards, and system health activities.
• Work closely with architects and technical leads on solution design and implementation.
• Continuously monitor and optimize system performance, scalability, and cost.
• Partner with Business Analysts to understand and implement integration requirements.
• Troubleshoot and resolve performance, backend, data, and functional issues.
• Analyze application and data-processing issues and identify root causes.
• Maintain strong attention to detail and document key technical decisions, solutions, and project artifacts.
• Support fast-paced projects and participate in troubleshooting and resolving production issues when required.
• Quickly learn and adopt new technologies, tools, and development methodologies.
• Communicate promptly and effectively with technical and business stakeholders.
• Participate in code reviews and provide technical guidance and mentoring to junior engineers.
• Collaborate with cross-functional teams to deliver reliable, scalable, and high-quality solutions.

Minimum Qualifications & Requirements

• Minimum 7 years of IT development experience.
• Strong experience developing data-driven backend solutions.
• Strong hands-on experience with:
• Python / PySpark
• Databricks
• SQL / PL/SQL
• Java or C#
• Backend development and Web Services/API integration
• Strong SQL skills with experience working with complex queries and data processing.
• Experience designing technical solutions and participating in architecture/design discussions.
• Experience performing code reviews and mentoring junior engineers.
• Strong analytical, troubleshooting, and problem-solving skills.
• Ability to investigate performance issues and identify root causes.
• Strong understanding of software development lifecycle, coding standards, and best practices.
• Ability to work independently as well as effectively within a team.
• Strong verbal and written communication skills.
• Must be a collaborative team player with a customer-focused mindset.

Preferred Qualifications

Experience with one or more of the following is highly desirable:

• Microsoft Azure
• Apache Kafka
• Node.js
• Azure Data Factory
• Cloud-based data engineering and integration solutions
• Distributed data processing and large-scale data platforms
• Monitoring, logging, and performance optimization tools