Big Data Engineer

Doran Jones; Tampa (FL); On-site; USD 80,000 - 100,000; Full time

Job summary

Doran Jones is seeking a Data Management professional based in Tampa, Florida. In this role, you will design and implement data architectures, develop ETL processes, and ensure data governance compliance.

The ideal candidate has over 3 years of data management experience, strong SQL skills, and familiarity with AWS tools. Effective communication and collaboration with cross-functional teams are essential.

Qualifications

• 3+ years of Data Management experience in an enterprise environment.
• 2+ years of experience with SQL.
• 1+ years of experience with AWS S3, Spark and Glue.

Responsibilities

• Design and implement scalable and robust data architectures.
• Develop and maintain ETL processes and data pipelines.
• Design data models and schemas for efficient data management.
• Integrate data from diverse sources ensuring seamless data flow.
• Optimize data pipelines for performance and efficiency.

Skills

Data Management

SQL

AWS S3

Spark

Glue

Education

Bachelor’s degree in Computer Science, MIS, or a related field

Job description

Responsibilities

• Data Architecture: Design and implement scalable, robust, and efficient data architectures, including data lakes, data warehouses, and data pipelines, to support the organization's data requirements.
• Data Pipeline Development: Develop and maintain ETL (Extract, Transform, Load) processes and data pipelines to ingest, process, and transform large volumes of data from various sources, ensuring data quality and reliability.
• Data Modeling: Design and implement data models and schemas to organize and structure data for efficient storage, retrieval, and analysis, considering business requirements and data governance principles.
• Data Integration: Integrate data from diverse sources, such as databases, APIs, files, and streaming platforms, ensuring seamless data flow and interoperability across systems.
• Data Transformation and Enrichment: Implement data transformation and enrichment processes to derive insights, perform aggregations, and create derived datasets for analytics, reporting, and machine learning.
• Data Governance and Compliance: Establish and enforce data governance policies, standards, and best practices to ensure data integrity, security, privacy, and compliance with regulatory requirements.
• Performance Optimization: Optimize data pipelines and queries for performance, scalability, and efficiency, leveraging techniques such as partitioning, indexing, and query optimization.
• Collaboration: Collaborate with cross-functional teams, including data scientists, analysts, software engineers, and business stakeholders, to understand data requirements and deliver data solutions that meet business needs.

Required Experience

• 3+ years of Data Management experience in an enterprise environment.
• 2+ years of experience with SQL.
• 1+ years of experience with AWS S3, Spark and Glue.
• Knowledge of metadata management, data governance, and data lineage.

Preferred Experience

• Good understanding of Agile Software Development practices and how to leverage these to streamline project execution.
• The ability to decompose business problems into smaller parts and solve them correctly, considering the impact on other subsystems.
• AWS certification.
• Team player with excellent problem-solving and troubleshooting skills.
• Familiar with software engineering practices, including CI/CD pipelines, Automated Testing, and Performance Engineering.
• Working knowledge of SDLC, especially regarding improving agile delivery metrics.
• Strong communication skills.
• Bachelor’s degree in Computer Science, MIS, or a related field.
• Financial services experience.
• Consulting experience.