So I told my students they were experts on this poem, "This is just to say"
They are all experts on human relationships. They are human beings, and relationships with other HB are fundamental to everything they do. This does not mean they are good at every relationship they have themselves, but they could analyze someone else's relationship with no problem.
So there several possibilities:
*passive-agressive: the speaker of the poem knows he's done something wrong, but it's ok, because he isn't going to say all the things she's done wrong to him.
*narcissism: he's a famous poet, so she's lucky to be with him. He can eat the damn plums if he leaves a poem for her.
*brother-sister. One student suggested that their relationship was a teasing kind of sibling relationship. He ate the plumbs, and gets to tease her too.
There is not wrong reading of the poem. You can be the guy who ate the plums or the woman who gets the note. You can feel whatever you want to feel. The poem is a poem, because it offers you these possibilities.