Neutron stars are probably not hiding dark matter under their skirts