You can only delete a set that is not used in documents and that is not linked to any product.

To delete a set: