The earliest reference to 'dog-ends' meant specifically cigarette ends found in the gutter, which down-and-outs could re-roll and re-use. The suggestion is of poor quality, just as it is in other uses of the word �dog' to mean a poor racehorse, an ugly woman and so on.
In technical terms - and this may be of more significance - a tool such as pliers, designed for gripping, has been called a 'dog' since the 15th century. Presumably, that was on the basis that actual dogs clamp things in their mouths. It might be, therefore, that the leftover end of a cigarette is the part that the smoker 'clamped in his mouth', as it were, whilst smoking it?
A further possibility is that it is a corruption of the word �dock' in the sense in which that word is used when referring to shortening an animal's tail. Thus, the thing might actually be a �dock-end'.