A few days of fasting for a healthy turtle should be no problem. You should still place more food than normal, you're turtle doesn't really have to fast. I'd suggest a bigger variety to what your feeding your turtle though. Red ear sliders are initially carnivorous turtles but they eventually turn mostly herbivorous. Even if you turtle is still young it might accept a carrot or some romaine lettuce here and there.
I'd definitely vote against taking him with you as that'd be too stressful for the turtle and it'll probably end up not eating at all throughout the weekend.
Before you're gone you need to keep in mind that there is no one to continuously take a look at the turtle in case something happens. Constant appropriate water depth and temperature are essential, you should keep a thermostat set at 78F or around, and if your tank is suspect to evaporation, i.e it has a big surface area, then add extra water. The water should never drop below the width of your turtles shell since if it does and your turtle flips on its back then that is a death sentence to your turtle if no one is around. If you're adding extra water and you have a powerful filter it's preferable you turn off the filter when you're not around so that the intake doesn't drown your turtle.
When you're gone dropping some feeder fish is a great idea, they won't rot away like most food would and it'd give your turtle a nice exercise. Avoid goldfish since theyre too high in fats and prevent vitamin B12 intake. If you can manage to attain some aquatic insects also then that'd be great, turtles absolutely adore them and that is mainly what many feed on when they're juveniles. Drop in a carrot peel, zucchini peel, etc.. Just don't overdo it so as not to dirty the water too quickly. You can also leave a few of your meal worms of course, and you can add some pelleted food, note the decreasing priority further down the list. If you have an automated feeder then you can add whatever you can fit there that your turtle eats and set it appropriately.
You will probably return to dirty water, which is why you need to clean up the minute you arrive. Have a lot of resting spots near the surface of the water so as your turtles never fatigues. Frankly, your turtle might even enjoy your absence since generally they're not big fans of being disturbed(unless it's for food). Your turtle will be able to take the dirty water provided you return at the time you provided, that's if your water dirties up(it depends on what you feed it when you're gone, and whether the filter is on or off). Have a great time when you're gone and try not to worry much about your turtle, take the appropriate measures and it'll do just fine. If you're still too worried you can have someone take care of it, feed it, or maybe just take a look at it to make sure all is going well.
If you have any other concerns or questions then please ask away.