I swatted a wasp when on my deck today and my dog ran over and picked it up, running away and chewing it up before I could catch him. Other than the potential of his being stung I was worried it may be toxic to him. He was a rescue and I don't know if this is something he did when running loose before the rescue group saved him. He was also observed by my daughter jumping, catching and ingesting a carpenter bee earlier in the day.

I am not a vet, but I do not believe that swallowing a dead wasp can hurt your dog provided he doesn't receive a sting in the mouth/throat area (which could cause problems if e.g., breathing was obstructed by swelling).

Those wasps and bees that sting are "toxic" in the sensee of having venom they can inject, but this is not generally very dangerous to mammals (leaving aside the obvious exception of allergies). My strong expectation is that the venom itself will be rendered harmless by exposure to stomach acid anyway.