Ben Gunn refused to go inside the fort because he had been living alone on the island for three years and was wary of being taken captive or mistreated by the pirates. He may have also wanted to avoid any potential conflict or confrontation with the pirates, as he knew they were ruthless and dangerous. Additionally, Ben Gunn may have believed that he would be safer and better remaining on his own, rather than risking his freedom and well-being by going inside the fort with the others.
