.net - Regex Replacing Non-words Chars In Strings, Ignoring Specific Chars?

In VB.net I've got the following line that removes all non-alphanumeric chars from a string:

return Regex.Replace(build, "[W]", "")

I now need to extend this to remove non-alphanumeric chars that aren't [] or _. I've changed the line to:

return Regex.Replace(build, "[W[]_]", "")

However I'm pretty sure that this says

replace non-word or [ or ] or _

how do I negate the tests for the [] and _ chars so that it says

replace non-word and not [ and not ] and not _

Some examples:

"[Foo Bar_123456]" => "[FooBar_123456]"
"[Foo Bar_123-456*]" => "[FooBar_123456]"