Senior Databricks Data Platform Engineer | AWS/ETL

Steampunk, Inc.; McLean (VA); On-site; USD 125,000 - 165,000; Full time

Job summary

Steampunk, Inc. in McLean, VA is seeking a Senior Data Engineer to design and implement enterprise-grade data platforms, pipelines, and services using Databricks. You will work with clients to deliver scalable solutions and communicate complex data concepts effectively.

Qualifications include 5-7 years in data engineering, experience with Databricks, Spark, Delta Lake, SQL, PySpark, and AWS. You will collaborate in an Agile environment, support data governance, and contribute to our AI & Data

Qualifications

• 5-7 years industry experience coding commercial software and a passion for solving complex problems.
• 5-7 years direct experience in Data Engineering with experience in tools such as Databricks, Spark, Delta Lake.
• Relational SQL experience (T-SQL preferred).
• Experience with Data Lakehouse architecture and Delta Lake/Apache Iceberg.
• Experience with data pipeline/workflow tools: Databricks Workflows, Airflow, Step Functions.
• Experience with AWS Databricks on AWS, S3, EC2, RDS.
• SDLC, CI/CD, and dev/test/prod environments experience.
• Knowledge of data cataloging tools (Informatica EDC, Unity Catalog, Collibra, Alation, Purview, DataZone) a plus.
• Commitment to data governance.
• Experience in Agile environments.
• Experience supporting teams of developers and data scientists.

Responsibilities

• Lead migrations of data using Databricks with focus on performance, reliability, and scalability.
• Assess ETL jobs, workflows, data marts, BI tools, and reports.
• Address inquiries concerning customization, enterprise architecture and feature functionality of data products.
• Experience with cloud data solutions in AWS (Databricks on AWS, S3, EC2, RDS).
• Support an Agile software development lifecycle.
• You will contribute to the growth of our AI & Data Exploitation Practice!

Tools

Databricks

Apache Spark

Delta Lake

Airflow

Step Functions

Informatica EDC

Unity Catalog

Collibra

Alation

Purview

DataZone

S3

EC2

RDS

Databricks Workflows

Job description

Steampunk, Inc. in McLean, VA is seeking a Senior Data Engineer to design and implement enterprise-grade data platforms, pipelines, and services using Databricks. You will work with clients to deliver scalable solutions and communicate complex data concepts effectively.

Qualifications include 5-7 years in data engineering, experience with Databricks, Spark, Delta Lake, SQL, PySpark, and AWS. You will collaborate in an Agile environment, support data governance, and contribute to our AI & Data