Rabbits in nature are only grabbed, if they are hunted down. Because of this they all have a strong aversion against being grabbed and carried around.
 
But if you do it gently and give your bunny time to became used to it, then it will be an unpleasant, but tolerable thing for your rabbit. You could help him by being calm during the handling and by giving him reward for it. The reward could be little amounts of the favorite food while he sits in your arms, or calm petting at his favorite spots. 

(Additionally, you could train to slow your heart rate. Like the most prey animals, rabbits are sensible for the heart rate of other animals nearby. If your heart stays calm, you give the rabbits the information that everything is good and there is no need for fear.)