Azure Data Engineer-Locals only

Skywalk Global; Indianapolis (IN); On-site; USD 100,000 - 160,000; Full time

Job summary

Skywalk Global is seeking a data engineer to support data integration, pipelines, data modeling, governance and documentation while collaborating with cross-functional teams to deliver high-quality, reliable data products for the agency.

The role plays a key part in implementing the agency’s data strategy by developing batch and streaming pipelines, optimizing data warehouse workloads, and enforcing governance standards.

Qualifications

• Master's degree in data science or related field, or 4+ years in data engineering
• Bachelor's degree + 2+ years rel./4+ years hands-on data engineering
• Certifications/courses may substitute for formal education/experience

Responsibilities

• Assist with the data technology strategy, including resolving strategic data integration issues such as data quality and stewardship, real-time
• Provide direction and oversight for the design and implementation of data solutions for integration and synthesis across multiple secure systems
• Assist with the definition, implementation, and continued adherence of new data management policies and design standards for data engineering
• Participate in project activities such as requirements gathering, solution reviews, and explaining technical complexities to non-technical personnel
• Work closely with the solution leads, project managers, data architects, data scientists, and data analysts on solution design, architecture, and implementation
• Assist with extraction, transformation, and loading of data from a wide variety of data sources using various data engineering tools and methods
• Apply data governance and security controls (Unity Catalog, RBAC, row/column security, data masking) to protect PHI/PII per HIPAA and agency policy
• Assist with documenting and demonstrating solutions by developing documentation, flowcharts, diagrams, and clear code comments
• Assist in operations by conducting systems analysis; recommending changes in policies and procedures
• The job description is not designed to cover all activities and responsibilities and may change at any time

Skills

Data integration

Pipeline development

Data modeling

Data governance

Strong communication

Requirements gathering

Documentation

Education

Master's degree in data science or related field

Bachelor's degree + 2+ years experience (or 4+ years hands-on data engineering)

Tools

Databricks

Auto Loader

Structured Streaming

Unity Catalog

Job description

Localcandidatesonlywillbeconsideredforthisposition.

Thisrolesupportsdataintegration,pipelinedevelopment,datamodeling,governance,anddocumentationwhilecollaboratingwithcross-functionalteamstodeliverhigh‑quality,reliabledataproductsfortheagency.

Playsakeyroleinimplementingtheagency’sdatastrategybydevelopingbatchandstreamingpipelines,optimizingdatawarehouseworkloads,andenforcingdatagovernancestandards.Thepositionassistswithrequirementsgathering,solutionreviews,andtechnicalsupport,andworkscloselywithdataarchitects,scientists,analysts,andprojectmanagerstotranslatebusinessneedsintotechnicaldesigns.Thisrolesupportsoperationalprocesses,contributestopolicyandprocedureimprovements,andmayprovidementorshiporsupervisionbasedonagencyneeds.

Job Duties

• Assistswiththedatatechnologystrategy,includingresolvingstrategicdataintegrationissues,suchasdataqualityandstewardship,real-time,
• Providesdirectionandoversightforthedesignandimplementationofdatasolutionsforintegrationandsynthesisacrossmultiplesecuresystems
• Assistswiththedefinition,implementation,andcontinuedadherenceofnewdatamanagementpoliciesanddesignstandardsfordataengineering
• Participatesinprojectactivitiessuchasrequirementsgathering,solutionreviews,andexplainingtechnicalcomplexitiestonon-technicalpersonnel
• Workcloselywiththesolutionleads,projectmanagers,dataarchitects,datascientists,anddataanalystsonsolutiondesign,architecture,andimplementation
• Assistswithextraction,transformation,andloadingofdatafromawidevarietyofdatasourcesusingvariousdataengineeringtoolsandmethods.
• Appliesdatagovernanceandsecuritycontrols(e.g.,UnityCatalogpermissions,role-basedaccess,row-andcolumn-levelsecurity,datamasking)toprotectsensitivehealthdata(PHI/PII)inaccordancewithHIPAAandagencypolicy
• Assistswithdocumentinganddemonstratingsolutionsbydevelopingdocumentation,flowcharts,layouts,diagrams,charts,codecommentsandclearcode.
• Assistsinoperationsbyconductingsystemsanalysis;recommendingchangesinpoliciesandprocedures.
• Thejobdescriptionisnotdesignedtocoverorcontainacomprehensivelistingofactivities,dutiesorresponsibilitiesthatarerequiredoftheemployee.Otherduties,responsibilitiesandactivitiesmaychangeorbeassignedatanytime.

Master'sdegreeindatascience,analytics,informatics,computerscience,mathematics,engineering,orrelatedfield;OR....Required4Years

Bachelor'sdegreeinarelatedfieldplus2+yearsofrelevantexperience;or4+yearsofhands-onexperienceinasimilardataengineeringrole;OR..Required4Years

Relevantcertificationsandcourseworkmaysubstituteforformaleducation/experiencerequirementsRequired4Years

DevelopingwithDatabricksnotebooksinPySpark,SparkSQL,andR,includingtransformations,joins,aggregations,andwindowfunctionsRequired4Years

Workingwithmedallion(bronze/silver/gold)architectureandincrementalingestionviaAutoLoader,StructuredStreaming,orLakeflow/DLTRequired4Years

UsingUnityCatalogforWorkspaceCatalogs,schemas,permissions,lineage,externallocations,andstoragecredentialsRequired4Years

Right-sizingclustersandSQLwarehousesanddiagnosepartitioning,shuffle,skew,andcachingissuesviaSparkUIRequired4Years

SchedulingandmanagingdependencieswithADFPipelines,DatabricksWorkflows(Jobs),andusingGitfolders(Repos)forsourcecontrolRequired4Years

DesigningMPPwarehousetables,includingdistribution(hash,round-robin,replicated),columnstoreindexes,andpartitioning

DevelopingwithCOPYINTO,PolyBase,andCETAS,andbuildingparameterizedSynapse/ADFpipelines(copy,dataflows,triggers)Required4Years

Workloadmanagement,resourceclasses,statistics,materializedviews,result-setcaching,andpoolscaling/pausingRequired4Years

MonitoringpipelinesandworkloadswithAzureMonitor/LogAnalyticsandalerting,andtroubleshootingfailedjobsRequired4Years

StrongwrittenandverbalcommunicationskillsRequired

Highlydesired