The rate at which things cool down has no relation to how they got hot. Don't try arguing. Female brains are wired differently and she'll tell you it stands to reason.
If all other things are equal something that starts of hotter stays warm longer. After all it must take a certain time to cool down to the temperature of something which starts at lower temperature. After that they both behave the same.
I don't think this is a straightforward issue. teh correct answer may be the counter intuitive one- for example why does hot water freeze more quickly than cool water?