The first thing I did- and this is very important- was to look at a lot of Myst art. Screenshots from the game, graphics from Myst-related websites, concept art, game guides, fan art... not only is it good research, allowing you to discover different shapes, colors, and effects that are common to the things you're trying to emulate, it also helps get you in the mood and mindset of the people whose style you're trying to mimic.
The first step in the actual design process was to set up my graphics. I created the images I wanted in Illustrator- at this point I did not worry about the colors or the jagged edges- I simply created the basic shapes. I made heavy use of the pathfinder tool- if you're not familiar with this you'll want to mess around with it a lot as it is incredibly useful.
I created the border image first, deciding to worry about the illustrations for the card last. This was just a matter of personal preference, it was what I felt like doing at the time.