Toilet training is all about positive reinforcement. It sounds like your puppy has got the hang of using a specific place to toilet nice and quickly, which is great. What you need to do now is get him to change where that place is. Toilet training requires a lot of patience and vigilance on your part, and consistency is the key. Whenever you are at home and with your dog, get into the habit of taking him outside on a regular basis and starting to use a key word to encourage to toilet. Like all training, he will not recognise this as anything at first, but whenever he does toilet outside upon you using said keyword, praise and reward him. He will start to associate toileting outside as a good thing. With puppies, it's always important to remember that they need to toilet a lot, just after they have eaten, just after they wake up, and almost every hour you are going to want to get him outside and go through the motions of using your keyword and really rewarding him.
With regards to changing what he has already learnt, when you are not home using crate can be a great toilet training aid, as dogs generally do not want to toilet where they sleep. If you are away for a short amount of time, say a few hours, shut him in the crate without the potty, and he should hold his toileting. As soon as you return, get him outside and use your keyword and rewarding. This will further enforce in his head that toileting outside is good. If you are away for longer than a few hours, you may want to leave the potty in with him as that is what he knows and you do not want him to have to hold it for too long.
The golden rule for toilet training is to never punish any accidents a dog has, you need to really work with your dog and aid him in making the decision to toilet outside.
In my personal experience working with dogs, toilet training pads or potties actually complicate matters, as like you said the dog becomes unsure where he is able to toilet. You basically want to teach him that going outside is always good, and going inside gets no reaction from you at all. Be prepared for accidents, as puppies are unpredictable, but only focus on the positive behaviours. You will also start to learn from your dog's body language when he needs to toilet; they will often look distracted and sniff the floor in circles, and sometimes head towards the door, so be alert at all times and work with your puppy, remembering to only focus on good behaviours.