Senior Data Engineer / Databricks

HTEC Group; United States; Remote; USD 120,000 - 160,000; Full time

Job summary

HTEC Group is seeking an experienced Data Engineer to enhance their data infrastructure. The ideal candidate will have over 7 years of relevant experience, strong hands-on capabilities with Databricks and Apache Spark, and proficiency in Python or Scala. Responsibilities include designing and maintaining data pipelines, ensuring data quality, and collaborating with business stakeholders on analytics. This position is critical to building scalable data solutions that facilitate insights and data-driven decision-making.

Qualifications

• 7+ years of relevant experience as a Data Engineer.
• Strong hands-on experience with Databricks and Apache Spark.
• Proficiency in Python or Scala, both preferred.
• Very good knowledge of SQL and relational databases.
• Experience with ETL/ELT principles.

Responsibilities

• Take ownership of data engineering features and code quality.
• Design, implement, and maintain Databricks-based data pipelines.
• Build and optimize ETL/ELT processes using Apache Spark.
• Ensure high data quality and accuracy across the platform.
• Collaborate with stakeholders to support data-driven products.

Skills

Data Engineering

Databricks

Apache Spark

Python

SQL

Tools

ETL/ELT tools

Cloud platforms (AWS, Azure, GCP)

Unix-like operating systems

Job description

We are looking for an ExperiencedData Engineer with strong hands-on experience in Databricks-based data platforms. The role focuses on building, optimizing, and maintaining scalable data pipelines, data lakes, and Lakehouse solutions that enable advanced analytics and data-driven products.

You will work on heavy data processing tasks, third-party integrations, ETL/ELT pipelines, and orchestration of data workloads in cloud environments. If you enjoy working with Databricks, Spark, large datasets, and modern cloud data stacks - and you are not constrained by a single programming language or tool - this role could be a great fit.

Key Responsibilities

• Take ownership of data engineering features, architecture, and code quality
• Design, implement, and maintain Databricks-based data pipelines and workflows
• Build and optimize ETL/ELT processes using Apache Spark on Databricks
• Design and manage data lakes and Lakehouse architectures (Delta Lake)
• Integrate diverse data sources and ensure reliable data ingestion
• Automate orchestration, scheduling, and monitoring of Databricks jobs
• Design and implement fault-tolerant and scalable data processing workflows
• Ensure high data quality, consistency, and accuracy across the platform
• Make informed decisions about storage, compute, and performance optimization
• Collaborate with analytics, BI, and business stakeholders to support data-driven products

Required Qualifications

• 7+ years of relevant experience as a Data Engineer
• Strong hands-on experience with Databricks and Apache Spark
• Proficiency in Python or Scala (both strongly preferred)
• Very good knowledge of SQL, relational databases, and data warehousing concepts
• Solid experience with ETL/ELT principles and data pipeline design
• Hands-on experience with cloud platforms (Azure, AWS, or GCP), preferably Databricks workloads
• Experience working with distributed systems and large-scale data processing
• Familiarity with Unix-like operating systems
• Experience with version control systems
• Strong communication skills and English language proficiency

Nice to have

• Databricks certifications - Professional level
• Experience with Delta Lake, performance tuning, and cost optimization
• Experience with streaming technologies (Kafka or similar)
• Knowledge of workflow orchestration tools (Databricks Workflows, Airflow, etc.)
• Experience with cloud-native and serverless data architectures
• Familiarity with containerization and virtualization (Docker, Kubernetes)
• Experience building data assets that directly support analytics and business decision-making