@jgoerzen nice post. In my mind the main problem with Matrix is metadata. Any of the federated servers your communication travels through has it - who, with whom, when.
Regarding Signal limitations and conversely P2P messengers needing both ends to be online simultaneously: Session addresses that. While not as well-rounded for normies, it's already easy enough to use and has most of the quality of life details that Signal does.