So, goldfish are glutinous fish and produce a lot of waste. As the comments above indicate, your tank size is too small. It is a common misconception that goldfish can live happy, healthy lives in small containment, like a bowl or small tank. They can live in small containers - but not in a happy and healthy way - unfortunately pop culture, like TV, movies, carnivals and unfortunately even some pet stores, show goldfish in bowls and this is only something their bodies can handle temporarily. The waste they produce causes a heavy build-up of ammonia, nitrates and nitrites, that make small containment basically a toxic environment for them, that can make them lame or depressed in personality and ultimately sick, and without proper care will lead to their death.
One good thing that makes things a little easier is that goldfish can withstand colder temperatures than tropical fish, meaning a heater is unnecessary, but because of how dirty they are they need filtration and weekly partial water changes.
The general rule of thumb is one gallon per one inch (around 1.5 L per 1cm) of projected growth of fish - so a 1.5 inch (3.8 cm) goldfish that gets to 8 inches (20 cm) needs at least 8 gallons (30 L) of water, and 2 of them means 16 gallons (60 L) of water, and so forth. If finances are an issue try finding a used tank at a yard sale or online, thoroughly clean it, and start from scratch by adding your existing water, filling the rest with treated fresh and adding a biological filtration additive like Cycle to Eco In a Bottle or Stresszyme and follow the instructions, sometimes daily or a week, and typically weekly from their forward for a month until your tank has gone through the nitrogen cycle.
Adding live plants and a filter will help keep your water clear, and will stay much cleaner and healthier than without a filter. I'm happy to see you have gravel, as that will help as part of your filtration once you have a bigger set up. You will need additional gravel, approximately 1 to 1.5 lbs per gallon of tank (0.12 to 0.18 kg per liter) and if your fish is still acting lame you may need to use this site to better diagnose an illness that may need additional treatment.
Also, once filtration is in place, invest in a cheap container of freshwater aquarium salt and follow instructions on the back based on your tank size. It will help to prevent some future scale and fin issues, add electrolytes to the water, and can later be used in larger doses as a sickness cure.
P.S. If you have done a water change and he seems a little healthier or more active, this is simply a temporary fix to the problem, and has given him some relief from the built up ammonia.
Also, having a light that's on during the day and off at night will help simulate a natural sun cycle, also healthier for fish activity.