Senior Databricks Engineer: Data Pipelines & Lakes

Svdigisol; United States; Remote; USD 120,000 - 180,000; Full time

Job summary

Svdigisol in the United States is seeking a data engineering specialist to design, implement, and manage scalable data pipelines on Databricks (PySpark, SQL, Delta Lake) hosted on AWS.

You will develop ETL/ELT workflows, build data lakes and warehouses, enable self-service analytics, enforce governance, collaborate with data scientists, tune performance, and maintain CI/CD pipelines with Terraform and Databricks CLI.

Responsibilities

• Design, implement, and manage data pipelines on Databricks (PySpark, SQL, Delta Lake) in AWS.
• Develop ETL/ELT workflows using Databricks, AWS Glue, Lambda, and Step Functions.
• Build and optimize data lakes (S3 + Delta Lake) and data warehouses (Redshift, Databricks SQL).
• Enable data democratization via Notebooks, APIs, and self-service analytics.
• Implement data governance, quality, and lineage with AWS Lake Formation or Unity Catalog.
• Collaborate with Data Scientists and Analysts to deliver ML-ready datasets.
• Tune clusters, jobs, and queries for performance and cost-efficiency.
• Manage CI/CD pipelines for data workloads (Terraform, GitHub/GitLab CI/CD, Databricks CLI).
• Ensure security, compliance, and best practices in data handling and access controls.
• Lead troubleshooting and incident management for large-scale data pipelines.

Job description

Svdigisol in the United States is seeking a data engineering specialist to design, implement, and manage scalable data pipelines on Databricks (PySpark, SQL, Delta Lake) hosted on AWS.

You will develop ETL/ELT workflows, build data lakes and warehouses, enable self-service analytics, enforce governance, collaborate with data scientists, tune performance, and maintain CI/CD pipelines with Terraform and Databricks CLI.