I think it's because when you actually get inside it, it looks nothing like how a functioning city should. It's all gothic arches and spires and balustrades and windows and statues as far as the eye can see, but almost no doors. Even the Undead Village resembles something of a real village, albeit a twisted one. Irithyll is like if an inhuman mind recreated a city solely based off of human sentiment.
And come to think of it, even the inhabited parts (the Church of Yorhska, the Chamber of the Princess, etc.) have largely been rendered decrepit and inhospitable by the sands of time.
The whole town has a very definite feeling of ornate inhospitability, only further compounded by the sanguinary abhorrences of Aldrich and company.