Data Engineer – Remote Databricks Pipelines

Irth Solutions; United States; Remote; USD 45,000 - 61,000; Full time

Benefits offered by this job

Competitive salary

Medical, dental, vision insurance

401(k) plan with company match

Generous PTO

Company-paid holidays

Flexible work options

On-call compensation

Job summary

Irth Solutions is seeking a mid-level Data Engineer to design, build, and maintain data ingestion and processing pipelines in Databricks, transforming external sources into clean, analysis-ready inputs for downstream intelligence. You will work primarily on the Stakeholder Engagement data and collaborate with Data Scientist and application teams.

The role emphasizes medallion architecture, Delta Lake optimization, reliability, and ongoing collaboration with cross-functional teams to deliver

Qualifications

• 3 to 5 years of experience in data engineering, with solid experience building and operating production-grade data pipelines.
• Familiarity with data modeling, data quality, and schema evolution.
• Strong proficiency in Python and SQL
• Hands-on experience with Databricks (Spark, Delta Lake)
• Experience with at least one major cloud (Azure preferred; AWS/GCP beneficial)
• Experience integrating with external APIs at scale: authentication, pagination, rate limiting, retries, error handling.
• Databricks certification preferred (Data Engineer Associate)

Responsibilities

• Data Pipeline Development (Primary Responsibility)
• Design, build, and maintain ingestion pipelines from high-volume external APIs, capable of running continuously and reliably at scale.
• Implement ingestion and transformation workflows using Databricks (Spark/PySpark, SQL, Delta Live Tables), applying medallion architecture patterns (Bronze → Silver → Gold) to move from raw ingested content to clean, structured, analysis-ready data.
• Build the infrastructure for deduplication and relevance filtering of incoming content, implementing filtering logic and quality criteria defined in collaboration with the Data Scientist.
• Implement schema evolution handling and data validation rules as data sources and formats change over time.
• Platform & Storage Implementation
• Configure and manage Delta Lake storage structures, tables, partitions, and optimization routines (OPTIMIZE, Z-ORDER, VACUUM).
• Design and evolve data schemas that balance query performance, cost, and maintainability as data volume grows.
• Maintain clear metadata and documentation of table structures to support easy consumption by the Data Science and application teams.
• Reliability, Monitoring & Operational Support
• Ensure pipeline reliability and observability: error handling, retries, monitoring, and alerting for a continuously running system.
• Adapt pipelines to evolving external API contracts, rate limits, authentication changes, and new data sources.
• Troubleshoot pipeline failures, perform recovery, and tune performance as needed.
• Orchestration & Automation
• Build, schedule, and monitor workflows using Databricks Workflows, Delta Live Tables, or similar orchestration tools.
• Contribute to CI/CD pipelines for code deployment, versioning, and environment management.
• Collaboration & Documentation
• Work closely with the Data Scientist to expose clean, well-structured data feeding LLM/NLP pipelines and downstream models.
• Participate in technical decisions around data architecture and propose structuring solutions as the team’s needs evolve.
• Document pipelines, data dictionaries, job schedules, and transformation logic.
• Support the onboarding of new data sources and pipelines as the product expands to additional solution areas.

Skills

Databricks

Python

SQL

Delta Lake

Spark

Data pipelines

CI/CD

Databricks Workflows

Tools

Delta Live Tables

Job description

Irth Solutions is seeking a mid-level Data Engineer to design, build, and maintain data ingestion and processing pipelines in Databricks, transforming external sources into clean, analysis-ready inputs for downstream intelligence. You will work primarily on the Stakeholder Engagement data and collaborate with Data Scientist and application teams.

The role emphasizes medallion architecture, Delta Lake optimization, reliability, and ongoing collaboration with cross-functional teams to deliver