What about introducing yet another global configuration array where people can register renderers?
Pseudo code:
$GLOBALS['TYPO3_CONF_VARS']['...']['Bootstrap'] => BootstrapRenderer::class.
In the view:
<f:flashmessages renderer="Bootstrap" />
I don’t see any better approach at this very moment to both let the user take control and keep the rendering out of the view helper.
Edit:
A little bit more sophisticated solution would be a one similar to the property mapping configuration, defining RendererResolvers with a priority and a method that evaluates whether the renderer should be used or not. I guess the current approach could easily be adopted. However I am not sure if such a solution makes sense or would be a bit over engineered then because I cannot come up with a case where I would let a renderer resolver decide when to use the bootstrap rendering or the list rendering.