Is there something to the idea that certain colors predators would avoid as it being synonimous of the fish being poisonous(even if that is not necesary the case).

I don't think brightly colored fish are poisonous.

There are a few studies which suggest that brightly coloured fish are more susceptible to predation:

https://www.google.co.uk/search?q=fish+ … p;ie=UTF-8