Senior Databricks Data Engineer - Spark, PySpark & AWS
Ethereum Technologies
A complete application in a minute
Tailored resume and cover letter, ready to send.
Job summary
Ethereum Technologies is seeking an experienced Senior Databricks Data Engineer to design, develop, and optimize large-scale data pipelines on the Databricks platform. You will work with PySpark, Spark SQL, and AWS cloud technologies to deliver enterprise analytics and regulatory-compliant data solutions.
The role involves building end-to-end ETL/ELT pipelines, integrating with AWS S3, Snowflake, Airflow, and AWS Glue, and implementing data quality, governance, and monitoring frameworks.
Qualifications
- 7+ years of hands-on experience with Databricks and modern data engineering platforms.
- Strong expertise in PySpark and Spark SQL.
- Experience building and managing ETL/ELT pipelines on cloud platforms (AWS).
- Knowledge of data modeling, data warehousing, and big data processing techniques.
- Experience with JSON, Parquet, Avro, CSV formats.
- Experience with data quality frameworks and monitoring solutions.
Responsibilities
- Design, develop, and maintain scalable data engineering solutions using Databricks, PySpark, Spark SQL, and Apache Spark.
- Build and optimize end-to-end ETL/ELT pipelines integrating Databricks with AWS S3, Snowflake, Airflow, and AWS Glue.
- Process and transform large-scale structured and semi-structured datasets for enterprise reporting and analytics.
- Develop reusable data ingestion, cleansing, transformation, validation, enrichment, and governance components.
- Tune Spark workloads to improve scalability and efficiency; troubleshoot production data pipelines and ensure high availability.
Skills
Tools
Job description
Ethereum Technologies is seeking an experienced Senior Databricks Data Engineer to design, develop, and optimize large-scale data pipelines on the Databricks platform. You will work with PySpark, Spark SQL, and AWS cloud technologies to deliver enterprise analytics and regulatory-compliant data solutions.
The role involves building end-to-end ETL/ELT pipelines, integrating with AWS S3, Snowflake, Airflow, and AWS Glue, and implementing data quality, governance, and monitoring frameworks.
'/%3e%3cpath%20d='M101.744%2012.4726L102.894%2016.1947L99.0021%2013.2201L101.744%2012.4726ZM105.3%208.40937H100.488L99.0026%203.59473L97.5116%208.41047L92.7%208.40385L96.5965%2011.3834L95.1055%2016.1942L99.0021%2013.2196L101.408%2011.3834L105.3%208.40992V8.40937Z'%20fill='white'/%3e%3cdefs%3e%3clinearGradient%20id='paint0_linear_5593_1381'%20x1='90'%20y1='9.44471'%20x2='99'%20y2='9.44471'%20gradientUnits='userSpaceOnUse'%3e%3cstop%20stop-color='%2300AD70'/%3e%3cstop%20offset='0.998594'%20stop-color='%2300B67A'/%3e%3cstop%20offset='1'%20stop-color='%23DCDCE6'/%3e%3c/linearGradient%3e%3c/defs%3e%3c/svg%3e){kind=small}
