Even the study of prosody needs to take into account the meaning of words. You wouldn't study prosody of a language you didn't know. You could, but you would be doing linguistics and not literature. To learn how Mallarmé empties language of meaning, you have to learn French first. Where learning French means learning signs in French, the combination of signifiers and signifieds. It follows that formalism and structuralism, and various kinds of "pure poetry," entail a kind of shift of emphasis toward the signifier, but the sign is still that bond between signifier and signified, even when fractured. It's a kind of pull in one direction, but a pull effected by semantic means. Usually you know that the poem aims to be autonomous from the world because the words of the poem are telling you that, through the meaning of those words, just like you know that a poem doesn't care about its own form because the words are telling you "I'm not one of those poems about roses and stuff like that, no, I am a poem meant to change the world." That poem could be just as beautiful, formally, as a poem that affects to turn its back on worldliness.
Even with prosody, it is the meaning of the words that are telling you how they are meant to sound. Sound doesn't imitate sense, rather sense tell you how to interpret sound. Formalism does lead back to a kind of motivated sign. Even in structuralism, there is a return of motivation (in Benveniste and Genette, for example).
Lévi-Strauss says that myth is translatable, in the way that poetry is not. It survives bad translation, because the structures are what matters, not intimate union between signifier and signified.