.Net Properly Shutdown An Asynchronous Socket Server Running On A Background Worker?

I have a Class called SocketSvr which handles an asynchronous socket server. It is called via the BackgroundWorker from my main form. Basically, I just want it to display my socket data information inside a textbox in the main form, and have a server start/stop button available on the main form to do those things.

Here is the code from Form1.vb:

Public Class Form1
Dim WithEvents Socketsvr As New SocketSvr
Private Sub ToggleServerButton_Click(ByVal sender As System.Object, _

[code]....

I am uncertain how to properly shut down requests from the client when the StopServer() function is run. In the code above, I place a cancellation queue for the background process. The funny thing is that once StopServer() is run, it will accept one more connection, then stop accepting thereafter.If I remove the following line from the code above:

listener.SetSocketOption(SocketOptionLevel.Socket, SocketOptionName.ReuseAddress, 1)

-- it crashes when I attempt to start the server a second time (complaining that I cannot reuse the socket, obviously) My guess is that I need to add something to the bw.CancellationPending call on the background worker?