Hi there, I recently found the answer, here. I have already solved it, use opencore to guide, and then use the vbiso to load the graphics card to drive the graphics card. hope this helps
https://www.insanelymac.com/forum/topic/345301-esxi-6770-amd-gpu-passhtrough/?do=findComment&comment=2780618