A very common thought that will come to mind when dealing with unwanted wild rodents in your dwelling are the very popular warfarin-like rodenticides (i.e. Rattax). Although I disagree a bit, but can't find my official sources to back me up, the concern about secondary poisoning for your cats are raised by the public. Although not cats so much, the biggest danger is primary ingestion by pets (mainly dogs) of these poisons. Rats can bump them off the shelves or carry them to more exposed areas. The humanity is also questionable, but see later as well.
In your situation, live traps are better. Commercially available, but you can also use, for example, one of those olden days 1L glass milk bottles placed at a +- 40 degrees angle and put some mouse treats in it. Place it where you see the droppings. The mouse will go in, but due to the slippery nature of glass won't be able to escape again.
The thing is, when relocating it to a nearby veldt or outside, being dwelling loving creatures, the mouse will either come back or very likely go and bother someone else, who might not be so kind.