It's probably not tansy; it's more likely to be ragwort. The caterpillars will be from a cinnabar moth (beautful, small, bright red). Although the moths are lovely, the ragwort should be removed - it's invasive and poisons some animals (not cats and dogs, though, as far as I know). And you shouldn't touch the caterpillars - they can give you a nasty rash.