Is the “poison” what makes it bitter?

That, I can’t say for sure.
But but considering there have been studies related to tolerance to different levels of tolerance to similar bitter foods, I’d say, you are close.

I remember one of my school teachers telling me about someone who died due to overconsumption of Neem leaves (he at around 15 - 20 a day). Now using an analogy: one can use Dettol™ antiseptic liquid to dress wounds, but you won’t go around bathing in the concentrated stuff right? It is a poison after all.

Why I can’t be sure: I haven’t studied it.
But I do know there are cases where some plant thing has a salient substance, which affects behaviour, but is not the nutritional part.
So, if you were to take that substance and put it in something else that’s bad for health, the subject (human/animal whatever) accustomed to that substance might attribute it to that original nutritional thing and eat the thing, but won’t get health benefits from it. At the same, if you give the original nutritional thingy sans that substance that is naturally in the plant part, the subject may not realise that it is getting the original nutritional value.

So, this case might be that the poison and the bitter thing are the same, but it also might not be.