Most modern houses have plenty of pre-fabricated elements in them anyway. (For example, carpenters no longer build roof structures; they're delivered to the site largely ready-made).
Further, the materials used the old pre-fabs wouldn't meet the thermal insulation requirements required from modern homes.
Regrettably too, there seems to be something about pre-fabs which can make people look down upon their occupants. I can't explain it and I'm certainly not seeking to justify it, but children who lived in pre-fabs were bullied mercilessly by the other children in the school I taught at.