From what I've seen, there's not a lot of settled science about the inner lives of fish. It's not really obvious whether they even experience emotional states like boredom in a way we would recognize. Most of the research in this field of animal behavior focuses on larger vertebrates from what I can tell.
When we talk about a bored pet, what we're basically talking about is whether the environment we provide offers enough stimulation to meet its behavioral needs. In other words, how do we recognize when we inadvertently cause stress by keeping an animal from interacting with its surroundings in the ways it's instinctively prepared for?
Like many animals, unusual behavior is one of the most visible indicators of stress in fish. Changes in behavior from things like poor water quality or disease are usually easy enough to spot, but the stresses we're concerned with here tend to cause more chronic problems. Identifying these means knowing what's typical behavior for the species, and recognizing when an individual is acting out of character. That could mean aggression in a peaceful fish, a heightened fear and startle responses in a calm one, or constant hiding in a bold one, for example. If a fish is known to readily breed in healthy conditions, it probably won't when it's under significant stress. The trick of course is that all these things can be caused by more urgent physical problems, but if your water parameters are good and no obvious disease presents itself over time, the stress may be psychological.
In many cases, it's not too hard to identify the source of the problem. A fish acting aggressively might need to be in a larger school, or it might need more hiding spaces it knows it can retreat to. These kinds of needs are often collectively known about your animal; if you have something kind of new to the trade, you may need to look at what's known about similar species, or even better, its natural behavior in the wild.
Generally this sort of thing is known as environmental enrichment, and my own opinion is that it will help minimize stress for most captive animals, aquatic or otherwise. Many aquarists provide some degree of this already. We keep community tanks, so there's plenty of inter- and intra-specific interaction. We keep plants and driftwood and other decorations that can be grazed on, hidden under, and claimed as territory. We (hopefully) provide enough physical space for the fish to be explore and be active within. All these things present our fish with regular "natural" events to initiate or respond to.
There are more proactive things we can do as well. Food in general is one of the best approaches we have: if you can, present it in a way that mimics how the fish would encounter it in the wild, like chasing down food in the water column or foraging in the substrate. (Fortunately, this is a big part of the branding for many prepared foods.) Live food that needs to be hunted is great, but just having a varied diet is helpful. Species that have distinctive reproductive behaviors, such as parental care or elaborate courtship rituals, should be encouraged to breed if you can accommodate it. Basically, look at the kind of puzzles your fishes are specialized to solve -- like "all my food lives in the sand" or "I need to defend my own little kingdom about 30cm in diameter" -- and try to provide safe opportunities for them to solve those puzzles. I suspect this is what the cause of "boredom" in pets, and even humans, ultimately boils down to.
Unfortunately the research I've found on this is mostly paywalled, and for the most part I can only get to abstracts. Here are some of the more interesting pieces:
This wiki article on fish intelligence has a lot of good information and references.