You're already ahead of the curve as you're thinking about this issue and measuring it instead just assuming that it's fine.
If possible, you should fix that one fat asset that's the bottleneck, and it sounds like other posters have given excellent advice here. However, if doing so starts to sop up lots of time, it probably is okay. Your user might even have the font cached already if it's common.
Fosstodon is an English speaking Mastodon instance that is open to anyone who is interested in technology; particularly free & open source software.