Deployment Of Encrypted App.config - Failed To Decrypt Using Provider

I've created a console .Net application and encrypted the app.config file with RSAProtectedConfigurationProvider. It works fine in an IDE on my XP desktop. Then I deployed my application to a Windows 2003 server. The package included the app's executable along with the encrypted app.exe.config file.

However, the application failed on the server with the following exception: Unhandled Exception: System.Configuration.ConfigurationErrorsException: Failed to decrypt using provider 'RsaProtectedConfigurationProvider'. Error message from the provider: The RSA key container could not be opened.

I decided to encrypt app.config on the server so I copied app.config and encrypted it. The app.exe.config looked encrypted and I ran the application. However, it failed: although it didn't raise an Rsa exception as before, it couldn't read a connection string. After I decrypted app.exe.config, I found that it contained only the below string

although the original app.config had contained an application-specific connection string. I think that in order to resolve my issue I should export the RSA keys from my desktop to the server. Can you advise how can I do it (it's not a Web application!).