Senior Data Engineer, AI & Databricks Pipelines (Remote)

United States Digital Space LLC; United States; Remote; USD 94,000 - 125,000; Full time

Benefits offered by this job

Health benefits

Life benefits

Voluntary lifestyle benefits

Job summary

Lumen is seeking a Sr. Software Engineer to modernize data platforms using PySpark and Databricks, accelerating AI enablement across cloud and on‑premises environments.

In a high-trust, agile setup, you will own design, development, and testing of scalable data pipelines, collaborate with partners, and contribute to governance, security, and operational excellence for AI‑ready connectivity. Remote work within the United States is supported, with emphasis on delivering reliable, compliant data

Qualifications

• 6+ years of experience with a bachelor’s degree (or 4+ years with a master’s) in Computer Science or related field; strong Databricks, PySpark, Spark, Python, SQL, and Azure Data Factory.
• Proficiency in PostgreSQL, Java, Spring Boot, REST APIs, and building AI-ready data pipelines with governance and analytics enablement.
• Experience with Kafka/event-driven architectures, Azure AKS, cloud networking, CI/CD, GitHub, Jira, and Agile delivery; adherence to engineering best practices.
• Strong collaboration, communication, documentation, problem-solving, and time-management skills; Bash/Unix scripting; Oracle RDBMS/PL/SQL preferred.

Responsibilities

• Engineer, design, code, test, debug, and support production-quality enterprise-grade systems in an Agile Scrum team.
• Support cloud migration and data modernization, including PySpark/Databricks data warehouse initiatives.
• Develop scalable, high-performance data processing solutions meeting SLAs for reliability and performance.
• Collaborate with business partners to meet governance, security, and compliance requirements.
• Support AI enablement and automation initiatives, including Databricks AI Genie where applicable.
• Build data engineering solutions using Databricks, PySpark, Python, SQL, Azure Data Factory, Java, Kafka, and Bash.
• Participate in code reviews and promote engineering best practices; document work for transparency and reuse.
• Prioritize and manage multiple concurrent projects and delivery commitments.
• Support regulated infrastructure environments handling compliance-sensitive workloads.

Skills

Databricks

PySpark

Apache Spark

Python

SQL

Azure Data Factory

ETL concepts

PostgreSQL

Java

Spring Boot

REST APIs

AI‑ready data pipelines

Kafka

Kubernetes

Bash/Unix scripting

CI/CD

Education

Bachelor's degree in Computer Science or related field

Master's degree in related field

Tools

Confluent Kafka

GitHub

Jira

Jenkins

Job description

Lumen is seeking a Sr. Software Engineer to modernize data platforms using PySpark and Databricks, accelerating AI enablement across cloud and on‑premises environments.

In a high-trust, agile setup, you will own design, development, and testing of scalable data pipelines, collaborate with partners, and contribute to governance, security, and operational excellence for AI‑ready connectivity. Remote work within the United States is supported, with emphasis on delivering reliable, compliant data