I started over with my kernel config from scratch. It went from 55 min, to 16 min and now only 10 minutes to compile the kernel. It's also considerably smaller in size. :)
Fosstodon is an English speaking Mastodon instance that is open to anyone who is interested in technology; particularly free & open source software.