Databricks Engineer

Cloud Analytics Technologies, LLC; Austin (TX); In loco; USD 150.000 - 190.000; Tempo pieno

Descrizione del lavoro

Cloud Analytics Technologies, LLC in Austin, TX seeks a Databricks Engineer to design and optimize scalable data pipelines on the Databricks Lakehouse Platform. You will build robust ETL/ELT processes, transform large datasets, and support enterprise analytics initiatives.

The role requires strong PySpark, Python, SQL, and Spark expertise, plus experience with Delta Lake, Unity Catalog, and orchestration tools like Airflow or Data Factory.

Competenze

• 5+ years of data engineering or big data experience.
• 3+ years with Databricks and Apache Spark.
• Strong Python, PySpark, and SQL skills.
• Experience with data ingestion, ETL/ELT pipelines.
• Knowledge of data modeling and data warehousing concepts.
• Familiarity with cloud storage (ADLS, S3, GCS).

Mansioni

• Design, develop, and optimize scalable data pipelines on Databricks.
• Process large datasets and implement data transformations.
• Collaborate with data architects and analysts to deliver analytics.
• Configure and optimize Databricks clusters and jobs.
• Implement data quality checks and governance standards.

Conoscenze

Databricks

PySpark

Python

SQL

Apache Spark

Delta Lake

ETL/ELT

Data pipelines

Formazione

Bachelor's in CS/IT/Data Eng

Master's preferred

Databricks Data Engineer Certification advantageous

Strumenti

Azure Databricks

AWS Databricks

Databricks SQL

Delta Live Tables

Unity Catalog

Airflow

Descrizione del lavoro

US Citizen, Green Card, H-1B, OPT-EAD, CPT, Other

Work Authorization: US Citizen, Green Card, OPT-EAD, CPT, H-1B, H4-EAD, Other

Position Overview:

The Databricks Engineer will design, develop, and optimize scalable data engineering solutions using the Databricks Lakehouse Platform. This role focuses on building robust data pipelines, processing large datasets, implementing data transformation workflows, and supporting enterprise analytics initiatives.

Technical Skills & Platform Expertise:

Strong hands-on experience with Databricks and the Lakehouse architecture.

Proficiency in Python, PySpark, and SQL for large-scale data processing.

Experience with Apache Spark, Spark SQL, and Delta Lake.

Knowledge of data ingestion, ETL/ELT pipelines, and data transformation frameworks.

Experience with Azure Databricks, AWS Databricks, or Databricks on Google Cloud.

Familiarity with cloud storage services such as ADLS, Amazon S3, and Google Cloud Storage.

Experience with Delta Live Tables, Workflows, Unity Catalog, and Databricks SQL.

Knowledge of data modeling, dimensional modeling, and data warehousing concepts.

Familiarity with orchestration tools such as Apache Airflow, Azure Data Factory, or Databricks Workflows.

Experience with Git, CI/CD pipelines, and automated deployment practices.

Understanding of data governance, access controls, security, and performance optimization.

Develop and maintain scalable data pipelines using Databricks and PySpark.

Build ETL/ELT workflows to process structured and unstructured data.

Implement data transformation and validation using Spark SQL and Python.

Design and manage Delta Lake tables and optimize data storage.

Configure and maintain Databricks clusters, jobs, and workflows.

Integrate data from multiple sources into centralized lakehouse environments.

Optimize Spark workloads, query execution, and cluster performance.

Implement data quality checks, monitoring, and error-handling mechanisms.

Apply Unity Catalog and governance standards to secure enterprise data.

Collaborate with data architects, analysts, and business teams to deliver analytics solutions.

Experience & Professional Capabilities:

5+ years of experience in data engineering or big data development.

3+ years of hands-on experience with Databricks and Apache Spark.

Strong programming experience with Python, PySpark, and SQL.

Understanding of distributed computing, data architecture, and performance tuning.

Strong analytical, debugging, and problem-solving skills.

Educational Background & Certifications:

Bachelor's degree in Computer Science, Information Technology, Data Engineering, or a related field.

Master's degree is preferred.

Databricks certifications such as Data Engineer Associate or Data Engineer Professional are an advantage.

We are an equal opportunity employer. All aspects of employment including the decision to hire, promote, discipline, or discharge, will be based on merit, competence, performance, and business needs. We do not discriminate on the basis of race, color, religion, marital status, age, national origin, ancestry, physical or mental disability, medical condition, pregnancy, genetic information, gender, sexual orientation, gender identity or expression, national origin, citizenship/ immigration status, veteran status, or any other status protected under federal, state, or local law.

Some of our Business Ventures - Knowledge Analytics

Do restrictive necessities, your capability for advanced analytics or your business model need custom-made applications and infrastructure configurations? opt for our hosted managed services to urge comprehensive analytics solutions supported industry-leading Cloud Analytics Technologies, LLC code - delivered by the Cloud Analytics Technologies, LLC consultants UN agency understand these solutions best. Through a mix of with-it technology and dedicated service and support, Cloud Analytics Technologies, LLC hosted managed services will facilitate you: