Senior Databricks/Python Developer

Compass

United States

Remote

USD 120,000 - 170,000

Full time

14 days+

A complete application in a minute

Tailored resume and cover letter, ready to send.

Get past ATS filters

Benefits offered by this job

Meal allowance
Health insurance
Dental insurance
Home office allowance
Welcome kit
Life insurance
Zenklub
TotalPass
Certification program with partners
Partnership discounts
Access to UOL content
Maternity kit
Extended maternity/paternity leave
Referral program

Job summary

Compass is seeking a data engineer to design and maintain scalable data pipelines using Databricks and PySpark. The role collaborates with engineering and data science to translate requirements into efficient transformations, cleanses data, and optimizes performance.

Proficiency in Pandas and XML ElementTree is required, with Azure cloud experience preferred. You will contribute to data architecture, ensure data quality through testing, and document data processes and pipelines for ongoing

Qualifications

  • Experience building data pipelines with Databricks (PySpark).
  • Experience processing and analyzing large datasets.
  • Proficient in Python data libraries (Pandas, PySpark).
  • Knowledge of XML data processing with ElementTree.
  • Experience with cloud platforms, especially Microsoft Azure.
  • Understanding of data quality, integrity, and monitoring.

Responsibilities

  • Design and develop scalable data pipelines using Databricks (PySpark) for processing and analyzing large datasets.
  • Collaborate with engineering and data science teams to understand requirements and deliver efficient solutions.
  • Implement complex data transformations, data cleansing, and aggregations using PySpark, and optimize code performance.
  • Use Python ElementTree for efficient XML data manipulation and heterogeneous data integration.
  • Develop Python scripts using Pandas for structured data manipulation and analysis.
  • Ensure data quality and integrity by implementing testing and monitoring practices.
  • Contribute to defining and implementing data engineering best practices and data architecture.
  • Maintain comprehensive technical documentation for implemented processes and solutions.

Skills

Python
Databricks
PySpark
Pandas
XML ElementTree
Azure
Data quality
Monitoring
Cross-functional collaboration
Data architecture

Tools

Databricks
PySpark
Python
Pandas
XML ElementTree
Azure

Job description

Responsibilities
  • Design and develop scalable data pipelines using Databricks (PySpark) for processing and analyzing large datasets
  • Collaborate with engineering and data science teams to understand requirements and deliver efficient solutions
  • Implement complex data transformations, data cleansing, and aggregations using PySpark, and optimize code performance
  • Use Python ElementTree for efficient XML data manipulation and heterogeneous data integration
  • Develop Python scripts using Pandas for structured data manipulation and analysis
  • Ensure data quality and integrity by implementing testing and monitoring practices
  • Contribute to defining and implementing data engineering best practices and data architecture
  • Maintain comprehensive technical documentation for implemented processes and solutions
Requirements
  • Experience developing data solutions using Python
  • Experience building and maintaining data pipelines with Databricks (PySpark)
  • Experience processing, transforming, and analyzing large volumes of data
  • Experience manipulating and processing data using Pandas
  • Knowledge of reading, processing, and integrating XML data using Python ElementTree
  • Experience with cloud platforms, especially Microsoft Azure
  • Knowledge of data quality, integrity, and monitoring practices
  • Experience defining and implementing data engineering and architecture best practices
  • Experience collaborating with Data Engineering, Analytics, and Data Science teams
  • Knowledge of performance optimization for pipelines and distributed processing
  • Familiarity with technical documentation for data processes, pipelines, and solutions
Benefits
  • Meal and/or food allowance
  • Health and dental insurance
  • Pet health insurance
  • Home office allowance (for remote positions)
  • Welcome kit during onboarding
  • Life insurance
  • Zenklub, with psychological and nutritional support
  • TotalPass
  • Certification and continuous training program with partners (Adobe, Anthropic, AWS, Google, Microsoft, Salesforce)
  • Partnerships and discounts to support your professional development
  • Access to UOL content
  • Maternity kit
  • Extended maternity and paternity leave
  • Referral program
Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Senior Data Engineer, Databricks
Senior Data Engineer, Databricks

Compass • United States

Hybrid
USD 140,000 - 200,000
Meal allowance
Health insurance
Pet health insurance
+6
JavaScript Front-End Developer – Mid-Level
JavaScript Front-End Developer – Mid-Level

Compass • United States

Remote
USD 90,000 - 140,000
Health and dental insurance
Home office allowance (remote)
Welcome kit for onboarding
+7
Tech Lead
Tech Lead

Jobgether • United States

Remote
USD 55,000 - 90,000
Meal allowance
Medical and dental insurance
Home office allowance
+3
Java Developer
Java Developer

Compass • United States

Remote
USD 120,000 - 180,000
Meal allowance
Health insurance
Home office allowance
+6
Senior AI/ML Developer
Senior AI/ML Developer

Compass • United States

Remote
USD 140,000 - 210,000
Meal allowance
Health insurance
Pet insurance
+8
Senior Full-Stack Developer, React, .NET
Senior Full-Stack Developer, React, .NET

BIP Brasil • United States

Remote
USD 90,000 - 140,000
Medical insurance
Dental insurance
Gym and wellness programs
+2
Junior Data Engineer (Databricks)
Junior Data Engineer (Databricks)

N-iX LTD • United States

Hybrid
USD 90,000 - 150,000
Flexible work format
Remote-friendly
Career development tools
+1
Senior Data Engineer Hot
Senior Data Engineer Hot

Binariks Inc. • United States

Hybrid
USD 120,000 - 180,000
Paid vacation
Public holidays
Sick leave
+10
Databricks/Python Developer Senior | Remote
Databricks/Python Developer Senior | Remote

Compass Uol • United States

Remote
USD 120,000 - 180,000
Vale alimentação
Plano de saúde
Plano odontológico
+8
Data Platform Engineer
Data Platform Engineer

ARCO • United States

Remote
USD 120,000 - 180,000
Food and/or meal allowance
Health and dental insurance
Transportation voucher
+7