No, it does not appear that anything harmful beyond an allergic reaction takes place.
If you search against "Bloodworm allergies", you will find many accounts related by people who have discovered that touching bloodworms (frozen or not) can cause allergic reactions, which is why you should wear gloves when handling them.
According to the study, "Fishing for Allergens: Bloodworm-Induced Asthma" at the Allergy, Asthma & Clinical Immunology (AACI) site:
Bloodworms, which are larvae of chironomids (also called freshwater midges), are often used as fish food or bait. Although chironomids can be found in Europe, North America, and parts of Asia, allergies to them are mainly due to their use as fish food. Individuals at particular risk of chironomid-induced allergy include fish farmers, fishermen, pet shop owners, and others who keep fish or work with fish [...]
There have been a number of previously reported cases of hypersensitivity to chironomids [...] . With the increasing popularity of aquariums, allergy to chironomids may become less of a novelty and become something clinicians should be aware of when searching for the cause of a patient's atopic symptoms.
Beyond the allergic reactions, which can be severe, I have found no other evidence of problems from touching bloodworms. Some of the messages I have read mention sensitivity increases over time, but I cannot find a study to back up that statement. It is probably best to avoid contacting them at all to avoid any severe reactions. I am not a doctor, therefore I cannot recommend experimenting with a mild touch to see if you have an allergic reaction.
The only relation with bacteria I can find is that using raw bloodworms can possibly transition bad bacteria to your tank; problems with touching bloodworms appears to be sourced solely from allergic reactions. It is possible these two separate concepts related to bloodworms (bacteria and allergies) may have mixed for you.
The warning from Hikari USA' Jumbo Blood Worms advises the following precautions:
Handling frozen food derived from a live animal can cause the same allergic reaction possible by handling the live animal itself. To avoid potential problems, always wash your hands immediately after use and avoid contact with your nose and mouth prior to washing.