Senior Databricks Architect: Data Pipelines & ML Excellence

ECS Corporate Services; Fairfax (VA); On-site; USD 175,000 - 250,000; Full time

Benefits offered by this job

Active TS/SCI-CI Poly Clearance

Job summary

ECS in Arlington, VA is seeking a Senior Databricks Consultant to design, optimize, and fine‑tune Databricks environments for advanced analytics, ML, and AI insights, leveraging Delta Lake, MLflow, and Databricks ML.

The role requires production Databricks experience, strong Python/SQL, Spark, and cloud platform skills (AWS/Azure/GCP), plus an active TS/SCI-CI Poly clearance. You will build ETL/ELT pipelines, ensure governance, and collaborate with data teams to deliver valuable datasets.

Qualifications

• Active TS/SCI-CI Poly clearance.
• Proven Databricks experience in production environments.
• Strong Python and/or SQL skills for data engineering.
• Experience with Apache Spark for distributed processing.
• Knowledge of data lakehouse architecture and best practices.
• Experience with cloud data platforms (AWS S3, Azure Data Lake, or GCP Storage).
• Hands-on experience building and managing ETL/ELT pipelines.
• Strong understanding of data modeling, data quality, and performance tuning.
• Familiarity with CI/CD pipelines and Git.
• Willingness to travel to customer sites; US citizenship required.

Responsibilities

• Design, build, and maintain scalable ETL/ELT pipelines in Databricks.
• Optimize workflows and clusters for performance and cost efficiency.
• Integrate Databricks with cloud platforms (AWS, Azure, or GCP).
• Implement data governance, security, and compliance standards.
• Collaborate with data scientists, analysts, and business stakeholders to deliver datasets and insights.
• Troubleshoot issues and drive continuous improvement in data infrastructure.

Skills

Databricks expertise

Python

SQL

ETL/ELT pipeline design

Apache Spark

Education

Bachelor's degree or equivalent

Tools

Databricks

MLflow

Delta Lake

Git

Cloud platforms (AWS/Azure/GCP)

Job description

ECS in Arlington, VA is seeking a Senior Databricks Consultant to design, optimize, and fine‑tune Databricks environments for advanced analytics, ML, and AI insights, leveraging Delta Lake, MLflow, and Databricks ML.

The role requires production Databricks experience, strong Python/SQL, Spark, and cloud platform skills (AWS/Azure/GCP), plus an active TS/SCI-CI Poly clearance. You will build ETL/ELT pipelines, ensure governance, and collaborate with data teams to deliver valuable datasets.