Databrick Engineer

Smart IT Frame LLC; Reston (VA); On-site; USD 90,000 - 120,000; Full time

Job summary

Smart IT Frame LLC is looking for a data engineer to design, build, and maintain data pipelines on the Databricks platform in Reston, Virginia. The role requires strong proficiency in Python and hands-on experience with Apache Spark and AWS services.

The ideal candidate will collaborate with cross-functional teams to optimize performance and ensure data quality, governance, and operational excellence across data platforms. This is an opportunity to work in a dynamic environment with current technologies.

Qualifications

• Hands-on experience with Apache Spark and big data processing frameworks.
• Strong proficiency in Python programming.
• Experience with AWS cloud services including S3, EMR, Lambda, SQS, SNS, and PostgreSQL.

Responsibilities

• Build scalable and reliable data engineering solutions on the Databricks platform.
• Create and maintain efficient ETL/ELT pipelines for data ingestion and transformation.
• Optimize Spark jobs and Databricks workloads for performance and cost efficiency.

Skills

Databricks

Apache Spark

Python programming

AWS services

Data quality assurance

Tools

PostgreSQL

S3

EMR

Lambda

SQS

SNS

Job description

Design, build, and maintain Databricks pipelines and infrastructure.

Develop, build, and deploy data pipelines and workflows using Databricks.

Develop and optimize Apache Spark applications for large-scale data processing and analytics.

Strong proficiency in Python programming.

Hands‑on experience with Apache Spark and big data processing frameworks.

Experience with AWS cloud services including S3, EMR, Lambda, SQS, SNS, and PostgreSQL.

Integrate Databricks with various data sources and enterprise platforms.

Implement data quality checks, monitoring, and validation processes.

Manage and configure Databricks environments, including clusters and notebooks.

Troubleshoot and resolve Databricks performance and functionality issues.

Collaborate with data scientists, analysts, and business teams to support machine learning initiatives.

Stay current with Databricks features, updates, and industry best practices.

Familiarity with data warehousing concepts and ETL processes.

Understanding of machine learning concepts and algorithms.

Strong analytical, troubleshooting, and problem‑solving skills.

Responsibilities:

• Build scalable and reliable data engineering solutions on the Databricks platform.
• Create and maintain efficient ETL/ELT pipelines for data ingestion and transformation.
• Optimize Spark jobs and Databricks workloads for performance and cost efficiency.
• Ensure data quality, governance, and operational excellence across data platforms.
• Support cloud‑based data architecture and integration initiatives.
• Work closely with cross‑functional teams to deliver data‑driven solutions.