Senior Data Engineer

ConsultNet Technology Services and Solutions; United States; On-site; USD 120,000 - 180,000; Full time

Job summary

ConsultNet Technology Services and Solutions seeks a Data Engineer to evaluate, design, and execute data structures, processes, and logic to deliver business value through enterprise data assets. You will work with subject matter experts to create scalable data assets using advanced data techniques and tools.

You will collaborate with data products teams to build pipelines, implement Spark-based transformations, and deliver robust data stores while ensuring data quality and governance across the

Qualifications

• 8 years of professional experience in data engineering.
• Hands-on with AWS and data platforms.
• Proficient in Python frameworks.
• Experience with Qlik Replicate and Snowflake.
• Expertise in building data ingestion pipelines and data quality testing.
• Familiarity with DevOps CI/CD, Docker, and Kubernetes.
• Background in Big Data Engineering and Advanced Data Analytics.

Responsibilities

• Design, build, and maintain data-based structures and systems.
• Create robust data pipelines for ingestion, transformation, and storage.
• Develop programs using Spark for data movement.
• Collaborate with data products and stakeholders to deliver usable data stores.
• Ensure data is prepared and ready for business use cases.
• Design and deploy microservices to serve data assets.
• Translate business needs into data designs and visualizations.
• Guide data strategy and aid in data product automation.
• Act as advisor to business leads on data opportunities.

Skills

Big Data Engineering

Advanced Data Analytics

Data Pipeline Design

DevOps CI/CD

Python Frameworks

Tools

AWS

Docker

Kubernetes

Qlik Replicate

Snowflake

Spark

Job description

Our client is seeking a Data Engineer who focuses on the evaluation, design, and execution of data structures, processes, and logic to deliver business value through operational and analytical data assets. The Data Engineer uses advanced data design and technical skills to work with business subject matter experts to create enterprise data assets utilizing state of the art data techniques and tools

Responsibilities

• Partners with Technology partners to Design, Build, and Maintain the data-based structures and systems in support of Data and Analytics and Data Product use cases
• Builds data pipelines to collect and arrange data and manage data storage in their big data environment
• Builds robust, testable programs for moving, transforming, and loading data using big data tools such as Spark.
• Coordinates design and development with Data Products Partners, Data Scientists, Data Management, Data Modelers, and other Technical partners to construct strategic and tactical data stores
• Ensures data is prepared, arranged and ready for each defined business use case
• Designs and deploys frameworks and micro services to serve data assets to data consumers
• Collaborates and aligns with technical and non-technical stakeholders to translate customer needs into Data Design requirements, and work to deliver world-class visualizations, data stories while ensuring data quality and integrity
• Provides consultation to all areas of the organization that plan to use data to make decisions
• Supports any team members in the development of such information delivery and aid in the automation of data products
• Acts as trusted adviser and partner to business leads- assisting in the identification of business needs & data opportunities, understanding key drivers of performance, interpreting business case data drivers, turning data into business value, and participating in the guidance of the overall data and analytics strategy

Required Skills

• 8 years of professional experience
• Amazon Web Services
• AWS Lambda
• Python Frameworks
• Qlik Replicate
• Snowflake (Platform)
• Build and test data ingestion pipelines - who understands ingestion pipeline engineering, query language, data integration tools like Qlik Replicate or precisely connect, and data quality testing source/target.
• Experience with DevOps principals and CI/CD.
• Experience with Docker and Kubernetes
• Background in Big Data Engineering and Advanced Data Analytics

Bonus Skills

• Prior banking or financial Services experience
• Experience developing solutions for the financial services industry
• Experience building data solutions at scale
• Experience designing and building relational data structures in multiple environments
• Experience with Airflow, Argo, Luigi, or similar orchestration tool
• Experience with No-SQL databases such as HBase, Cassandra, or MongoDB
• Experience with streaming technologies such as Kafka, Flink, or Spark Streaming
• Experience working with Hadoop ecosystem building Data Assets at an enterprise scale
• Proven record of accomplishment of delivering operational