I know that somewhere out there from long ago, there is a set of old icons which by now are incredibly outdated in both the areas of art style, quality, and the characters.
This set contains nearly all of the characters (exclusions below) in both a .png and .ico file format. All of them match each other, but in the .png folder particularly, some are higher quality and larger than the others. The preview image shows the difference (Nordics were used because the Allies all have the same quality as the Axis). All art is official.
Download the .zip file in the DeviantArt download link on this page.
How to change icons varies between computer. And by that I mean, just google it or right-click what you want to change and hunt through the options and properties.
Not included: Romania, Bulgaria, Kugelmugel, Molossia, Nikoniko, Cyprus, Macao, Ladonia, TRNC, Korea, India, Hutt River, Hesse, Moldova, and the other even more obscure ones.