Chaffing on one side only?

So I looked at my bibshorts the other day and noticed that i'm getting rubbing/fraying on the material on 1 side only.

what can be the cause of this? Saddle too high? I've tried lowering it in the past but really didnt like how it felt.