That's not progress being removed, that's because new items are added to the pool inside the Apparel Caches.
So you previously had already received 97 out of the 100 items that could drop from the caches. Now, with the new Season, some items were removed from this pool and new items were added to the pool instead. Which basically means that no items have been removed from your inventory, instead you have been given the chance to collect more new items thanks to the new loot pool.
Or in other words: the "97 of 100" does not refer to your inventory (you should by now own waaaaay more apparel pieces than only 97) – it is referring to how many of the items inside the loot pool of the Apparel Caches you already have inside your inventory.
Example:
Let's say the caches of Season 5 contained the letters A, B, and C. Since you opened many caches during that Season you now own the letters A, B, and C. Thus your progress is being shown as "3 of 3". Now, with Season 6, the letters B and C get removed from the caches while the letters D and E are being added instead. Thus your progress is now being shown as "1 of 3". Because you already own A from the previous Season (which is still inside the loot pool) but you do not own D or E yet.