Senior Data Engineer

Plan A Technologies, Inc.; Las Vegas (NV); Remote; USD 120,000 - 160,000; Full time

Job summary

A leading technology advisory firm is seeking a Senior Data Engineer with expertise in Databricks and extensive experience in data engineering. The role involves designing data pipelines, implementing data governance, and optimizing performance, while offering a unique work-from-home flexibility with a supportive environment.

Benefits

Generous vacation schedule

Brand new laptop

Supportive work environment

Qualifications

• 6+ years of experience in data engineering.
• Strong focus on Databricks and big data processing.
• Experience with metadata frameworks.

Responsibilities

• Design, develop, and maintain scalable data pipelines.
• Implement Unity Catalog for data governance.
• Analyze cluster logs for performance optimization.

Skills

Databricks

SQL

PySpark

Problem-solving

Education

BS in Computer Science

Job description

Plan A Technologies is looking for a highly skilled Senior Data Engineerwith extensive experience in Databricks, including Unity Catalog for data governance. The ideal candidate will possess a strong ability to analyze and optimize cluster configurations using cluster logs and demonstrate a proactive approach to independently drive tasks to completion.

Please note:you must have at least 6+ years of experience in data engineering, with a strong focus on Databricks and big data processing to be considered for this role.

JOB RESPONSIBILITY

Design, develop, and maintain scalable data pipelines and workflows within the Databricks environment, ensuring optimal performance and cost-efficiency.

Unity Catalog Implementation:Deploy and manage Unity Catalog to provide centralized data governance, access management, and auditing across Databricks environments.

Analyze cluster logs to fine-tune configurations, optimize resource allocation, and enhance overall system performance.

Take ownership of well-defined tasks, driving them to completion with minimal supervision, and proactively addressing potential challenges.

Work closely with cross-functional teams, including data scientists, analysts, and other engineers, to understand data requirements and deliver robust solutions.

Maintain comprehensive documentation of data workflows, configurations, and best practices to support knowledge sharing and onboarding.

EXPERIENCE

BS in Computer Science, Software Engineering or equivalent practical experience

6+ years of experience in data engineering, with a strong focus on Databricks and big data processing. Knowledge of the gaming/financial industry is a bonus.

Extensive Databricks experience, able to diagnose and significantly improve performance for complex Spark workflows.

Comfortable with a metadata framework.

Extensive experience with PySpark and with Unity Catalog.

Strong knowledge of cluster configuration and performance tuning.

Documents code and/or produces solid documentation, implementing it using best practices.

Significant experience with SQL, able to edit complex queries for performance and quality, and to conduct and automate data validation. Automated testing is a bonus.

Experience implementing automated data quality, validation, and code testing procedures

Nice to have: Certifications related to Databricks or cloud platforms.

Self-starter/independent worker. Demonstrated ability to work independently and take initiative.

Excellent verbal and written English communication skills.

Excellent problem-solving and analytical skills.

Strong communication and interpersonal abilities.

Initiative and drive to do great things.

ABOUT THE COMPANY/BENEFITS

Plan A Technologies is an American software development and technology advisory firm that brings top-tier engineering talent to clients around the world. Our software engineers tackle custom product development projects, staff augmentation, major integrations and upgrades, and much more. The team is far more hands-on than the giant outsourcing shops, but still big enough to handle major enterprise clients.

Read more about us here: www.PlanAtechnologies.com .

Location:Work From Home 100% of the time, or come in to one of our global offices. Up to you.

Great colleagues and an upbeat work environment:You'll join an excellent team of supportive engineers and project managers who work hard but don't ever compete with each other.

Benefits: You’ll get a generous vacation schedule, Brand New Laptop, and other goodies.

JOB OVERVIEW

Plan A Technologies is looking for a highly skilled Senior Data Engineerwith extensive experience in Databricks, including Unity Catalog for data governance. The ideal candidate will possess a strong ability to analyze and optimize cluster configurations using cluster logs and demonstrate a proactive approach to independently drive tasks to completion.

Please note:you must have at least 6+ years of experience in data engineering, with a strong focus on Databricks and big data processing to be considered for this role.

JOB RESPONSIBILITY

• Design, develop, and maintain scalable data pipelines and workflows within the Databricks environment, ensuring optimal performance and cost-efficiency.

• Unity Catalog Implementation:Deploy and manage Unity Catalog to provide centralized data governance, access management, and auditing across Databricks environments.

• Analyze cluster logs to fine-tune configurations, optimize resource allocation, and enhance overall system performance.

• Take ownership of well-defined tasks, driving them to completion with minimal supervision, and proactively addressing potential challenges.

• Work closely with cross-functional teams, including data scientists, analysts, and other engineers, to understand data requirements and deliver robust solutions.

• Maintain comprehensive documentation of data workflows, configurations, and best practices to support knowledge sharing and onboarding.

EXPERIENCE

• BS in Computer Science, Software Engineering or equivalent practical experience

• 6+ years of experience in data engineering, with a strong focus on Databricks and big data processing. Knowledge of the gaming/financial industry is a bonus.

• Extensive Databricks experience, able to diagnose and significantly improve performance for complex Spark workflows.

• Comfortable with a metadata framework.

• Extensive experience with PySpark and with Unity Catalog.

• Strong knowledge of cluster configuration and performance tuning.

• Documents code and/or produces solid documentation, implementing it using best practices.

• Significant experience with SQL, able to edit complex queries for performance and quality, and to conduct and automate data validation. Automated testing is a bonus.

• Experience implementing automated data quality, validation, and code testing procedures

• Nice to have: Certifications related to Databricks or cloud platforms.

• Self-starter/independent worker. Demonstrated ability to work independently and take initiative.

• Excellent verbal and written English communication skills.

• Excellent problem-solving and analytical skills.

• Strong communication and interpersonal abilities.

• Initiative and drive to do great things.
  
  

ABOUT THE COMPANY/BENEFITS

Plan A Technologies is an American software development and technology advisory firm that brings top-tier engineering talent to clients around the world. Our software engineers tackle custom product development projects, staff augmentation, major integrations and upgrades, and much more. The team is far more hands-on than the giant outsourcing shops, but still big enough to handle major enterprise clients.

Read more about us here: www.PlanAtechnologies.com .

Location:Work From Home 100% of the time, or come in to one of our global offices. Up to you.

Great colleagues and an upbeat work environment:You'll join an excellent team of supportive engineers and project managers who work hard but don't ever compete with each other.

Benefits: You’ll get a generous vacation schedule, Brand New Laptop, and other goodies.

If this sounds like you, we'd love to hear from you!