I think a lot of the anger and suspicion towards the Speaker came out of a sense of a frustration towards the original game’s campaign (you’d think it’d be towards the Exo Stranger, really), a hunger for conspiracy theories, and a sort of edgy hip dislike of anything like religion. He also didn’t say much, but as The_Darkness said, his VA was pricey. Bit of a Dinklage situation, I imagine, with less need to replace him.
I, too, don’t think there’s anything in lore that supports those attitudes. The Speaker’s role seemed pretty clearly defined to me. He provided spiritual guidance and a sort of political ‘let’s stay on track’ guardrail. Even with The Traveler awake, there would still be need of this. His replacement should not be left as a dangling thread.
(Edited for spelling. Cold medicine, ladies and gentlemen, cold medicine.)