Delete Invisible Text From Richtextbox?

I'm using a richtextbox which contains some invisible text; this is so I can have the user add a hyperlink - the code inserts the hyperlink as invisible RTF text after the link text. The problem I have is trying to delete the hyperlink - if I try and set the selectionstart and selectionlength to select the hyperlink, they ignore the invisible part (which is all of it).

So, how do I select the invisible text so it can be deleted? - I have the appropriate index and length available. An alternative approach is to somehow map the index to find the location in the rtf property of the RTB and delete the text in there;