Depending on how the electronics work, it's possible that it may damage the output stage of the amplifier. Put simply, if the amp is producing wattage/power onone channel but the wattage has nowhere to go (because the speaker is not connected) you could "burn out" that channel of the amp. If you turn the balance control all the way to the connected channel, that may be "safer" (although, not having much electronics know-how, I can't say for certain) but you'll only be hearing half of the music! Best bet is to re-instate the speaker, even if it's not positioned precisely where it should be.