What is this white-ish buildup on my fish filter? Should I be cleaning more often?
I have never had an external filter like this and I have never experienced such a buildup in the filter - I have always been using internal, fully submerged filters - but based on your photo and description of the material being slimy I would say it is a bacterial biofilm that is attached to the precipitated limescale underneath.
I do not think it is a definitive sign that the filter is dirty nor that it needs immediate cleaning; after all, it is a natural and even desired for the filter to build up the beneficial bacterial flora and the porous structure of precipitated limescale is an attractive place for them to colonize. Limescale buildup itself may be an unsightly nuisance, but is more or less unavoidable in most cases. I do not think that the flora inhabiting this limescale precipitation is large enough to contribute in any significant way to the overall flora of beneficial bacteria in your aquarium, especially that nitrifying bacteria are photosensitive and this material is placed outside where it is exposed to light - but nonetheless, it theoretically increases the efficiency of your filtration a little.
However, it definitely should be cleaned once you notice that it starts to block water flow, as that would decrease the efficiency of your filter. As long as it doesn't, I'd say that it is harmless or even mildly beneficial.
Other than that, I think it is reasonable to assume that this precipitated material is harmless because its mineral content comes from what was originally dissolved in the aquarium water itself.
Nonetheless, please consider that my reasoning above in regards to the nature of biofilm might be wrong and its nature - as suggested by Trond Hansen in the comment - is not bacterial, but fungal. If this is indeed fungal growth, then it should be definitely removed because some fungi are known for releasing potent toxins.
It will be helpful for determining the nature of this growth if you provide results of pH, GH and KH for the aquarium water. In general, slightly acidic environment favors fungal growth, whereas neutral to slightly alkaline conditions favor bacterial growth; there are exceptions, though.