I don't think more paving is a good idea. It could make your garden vulnerable to flooding in periods of heavy rainfall because there will be nowhere for the water to drain away. The moment you put soil down and plant more shrubs, you will immediately put yourself into an endless cycle of weeding and pruning. Have you thought about removing the gravel and covering it with forest bark. This is a substance that will enable drainage, it won't get dirty like gravel and in due course it will slowly decompose and you can top it up with more. If you think this will look stark on its own, invest in two or three very large decorative pots and plant them with some variegated evergreen shrubs to add all year round colour and interest. . Also if you ever have to move house and somebody else wants to do something with the garden, the area covered by forest bark will be easily deal with, without large areas of paving stones or concrete having to be dug up and removed.