Search For Text BackColor In A Rich Text Box?

I'm trying to make a program that can do some statistics on text in a rich text box. I'd like to count the number of instances where text appears with a certain back color, for example where text is "highlighted" with yellow.

There seems to be no function to get the backcolor of text at a specific index, you can only get the backcolor of text which is selected, like this:

Dim backColor As Color
backColor = RichTextBox.SelectionBackColor

This means, by applying this nasty trick, I would have to manually select every character in the rich text box before I can find the backcolor:

For i = 0 To RichTextBox.Text.Length
RichTextBox.Select(i, 1)
If RichTextBox.SelectionBackColor = Color.Yellow Then
yellows = yellows + 1
End If
Next

However, this takes a very long time, because the program has to select and deselect every character.

What options are there to speed up the process? Disabling the paint event?