Databricks Data Engineer: Modernize ETL & Pipelines

Infinityquest It Services; United States; Remote; USD 90,000 - 130,000; Full time

Job summary

Infinityquest It Services is seeking an experienced Data Engineer to help modernize our data platform by analyzing existing ETL pipelines and migrating them to a Databricks-based platform. You will design scalable data pipelines for batch and near real-time processing and work with AWS services.

The role requires strong Python/SQL skills, Spark knowledge, and hands-on Databricks experience. You will collaborate with analysts and data teams to deliver reliable datasets and maintain observability.

Qualifications

• Experience building and migrating ETL/ELT pipelines.
• Proficiency with PySpark and SQL.
• Experience with AWS data services.
• Familiarity with Databricks monitoring tools.

Responsibilities

• Analyze and migrate existing ETL pipelines to Databricks.
• Design, build, and maintain scalable data pipelines for batch and near real-time processing.
• Implement CI/CD pipelines for data workflows.
• Ensure data quality, governance, and logging.
• Collaborate with stakeholders to define data requirements.
• Monitor pipelines and resolve data issues.
• Optimize for cost, performance, and scalability in cloud.

Skills

SQL

ETL pipelines

Python

PySpark

Spark

Databricks

AWS

Data modeling

Data warehousing

Data Lake

Tools

Databricks

Delta Lake

Notebooks

Jobs

Workflows

CloudWatch

Job description

Infinityquest It Services is seeking an experienced Data Engineer to help modernize our data platform by analyzing existing ETL pipelines and migrating them to a Databricks-based platform. You will design scalable data pipelines for batch and near real-time processing and work with AWS services.

The role requires strong Python/SQL skills, Spark knowledge, and hands-on Databricks experience. You will collaborate with analysts and data teams to deliver reliable datasets and maintain observability.