tourist trap

Well, actually, the pillars each infused a crystal seed with their magic, and over time, these seeds became the Tree of Harmony.

Also, of course plant perception is a thing. If one plant is attacked by insects, for example, nearby plants with start producing substances to fight these insects, even if they themself are not yet under attack.
Therefore, if the castle, which is connected to the tree, which is connected to all of Equestria, senses that something is threatening the land, it´ll do what it can to procte the land.
Sure, it won´t stand up and kick some flank, but it can use the table-map-thingy to summon ponies or other magical beings, such as Spike, to certain locations.

The Tree of Harmony was created from the light of Equestria, a. k. a. the Pillars, and the seed it created, which grew into the castle, most likely needs this light to survive, just like the tree itself. Therefore, if the Tree of Harmony notices a threat, it lets the castle know, which then sends the most qualified beings in its reach to eliminate this threat. That´s basic plant immune system behavior, and if plant perception was not real, this wouldn´t work.

Also, Equestria is not our world, so I guess we can assume that not everything has to be the exact same in both worlds. I mean, in a world inhabited by magical, talking ponies, shape-shifting insects and competent political figures, is it really so unlikely that a castle, which has magically grown out of crystals formed from the piece and harmony of the realm I may add, posesses enough sentience to detect threats to its life, and then send other beings to deal with this threat?

Anyway, if the castle were a crystalized plant, this would mean that it came to be when a plant turned into a crystal, which is not the case. You see, the castle is a crystal, created by the Tree of Harmony, which got strengthened by the Elements of Harmony being returned to it after they themselves were strengthened by the Mane 6. The Tree then created the seed/box/whatever you want to call it, and when the Mane 6 used their keys, representations of their understanding of the Elements of Harmony, the box got the strength to grow into the castle.



Ok, this turned out a bit longer than intended. Anyway, thanks for reading.

Because it's a show that isn't afraid to tackle serious subjects?