Why any man would have a special connection to an article of clothing is beyond me. Unless it belonged to a deceased family member, it’s just a t-shirt. (Say the same thing to me, however, and there’s trouble.) The important question here is whether or not you care that those pajama pants were “special.”
Do you care?