First hedges grow much denser than rose bushes, and thorns are not an effective deterrent for a dog. Even a fence is not always a good means of containing a dog, who is determined to leave. I have new neighbors who moved into house with a 40 year old hedge as a fence, it was about a week before the first dog went through the hedge, and their dogs are always supervised when they are outside.
These related questions discuss alternatives to fences
There are several options for using plants to disguise a fence, but these will require first a fence appropriate for containing your dog, and then years for the plants to grow sufficiently to cover the fence.
I looked for studies about containing dogs with plant growth and came up empty. I did find a 1845 work in Blackwood's Edinburgh magazine, Volume 57, that talks about dogs and hedges as fences, while it does not give much information about keeping a dog in, it does speak abut the years required to make a hedge to contain livestock (which is easier than containing a dog).
If you have a dog who is too timid to test boundaries. Who might possibly be contained by a minimalist boundary divider, and if that dog is EVER going to be unsupervised in the yard, than you need a fences capable of keeping out all of the neighborhood dogs and wild life that can harm your dog.
For small dogs see the related question https://pets.stackexchange.com/questions/6785