We think of music and poetry as contiguous spaces. They aren't the same thing, but there is a passageway between them. So the number of poems that can be set to music (that I can imagine setting) is a rather small percentage of poems I like in other ways. And if I write a tune first, I find it extremely difficult to come up with a satisfactory lyric, so the passageway is narrow in both directions.
Imagine it were different? I suppose if I was working in a poetic genre already that was tin pan alley or madrigal, then the tunes would come easily.