What happened to the Imperial remnants?

Besides the First Order, we know that some territories in the core world and outer rim were left to the Galactic Empire after the signing of the Galactic Concordance. Though they are just a few sectors and no danger to the republic, this is what’s left of the Empire in Canon after 5 ABY. What happened to these territories? Were they conquered?

We currently have very, very limited knowledge of what the political state of the galaxy was between RotJ and TFA. To the best of my knowledge, the few remaining territories were essentially in a cold war with each other and with the New Republic, though it doesn't seem like open war kicked off again between the signing of the Concordance and the attack by the First Order in TFA.

Hopefully that gap will be filled soon

It is currently being filled my the Mandalorian, and possible more to come

DarthCaedusDad came in and destroyed them along with his second family.

Darth Caedus Dad = Han Solo, so incorrect

It’s a Slorg joke EC