VS 2010 : Working With Large Amounts Of Data Without Getting Out Of Memory?
Apr 25, 2011
I recently wrote a program that collects information from the network can displays it in a DGV for the user to see. I tested this on networks where I would end up with several thousand items brought back and all worked fine, very little memory increase and once the search had finished the memory usage went back to pretty much what it was before the search, so I assume I have no major memory leaks that I should be concerned about (I'm careful to always dispose of things and clear large variables where possible).
However, a new company has now started using my application and they have several hundred thousand items that my program will find. They reported that they leave the program running its search for a few hours and most of the time it crashes with an Out Of Memory exception. Now I did have my app do some caching of certain information to avoid querying the network more than was necessary so I thought well maybe this is what is causing it but I added an option to turn that off and they say it still happens. So the only thing I can see it being is simply the amount of data that the program is finding and having to store in memory.once my application has gathered information about the items it stores them in a List(Of MyItemClass) and then when the search is complete the items are added to the DGV. So I'm thinking maybe its just the size of this List(Of MyItemClass) that is simply getting too large. Bear in mind that each instance of MyItemClass in this List can have quite a lot of information in it as it has several properties that are List(Of String) that may contain a few thousand strings in some scenarios.How do you deal with such large amounts of data without running out of memory?I'm assuming the only option I have got is to 'page' some of the data to a temporary file on disk once I get over a certain number of items in the List(Of MyItemClass)?
Is it possible to create a kinda virtual storage in a vb application so that data created during run time are stored right in the applicaion instead of its environment?
What I want to do is display large amounts of text and a few pictures, ideally with hyperlinks between the pages. A lot like wikipedia really, lots of similarly arranged pages, text in categories with in-line hyperlinks and the occasional picture to illustrate a point.But not online, integrated as part of a larger app.
How would it be best to go about doing that? The display method is sortof dictated by the storage method. My thoughts were:
Lots and lots of labels stored in an Access database
Straight HTML as if it were a webpage embedded in an iFrame or something
Labels populated from XML files?
Program the whole lot in Flash or something else (A language I can't speak)
This is only a mockup for demonstration purposes so the priority is user-friendlyness and ease of programming rather than storage concerns or good programming form.
if I want to submit a large amount of text to my asp web page, do I need to worry about the text itself such as containing "&a=" that would mean another variable is being assigned for submitting data? Any way of not worrying about this? Or do I need to replace the & symbol everywhere in the test first if used?
I need to constantly display large amounts of colored text in WPF or Silverlight, and I need the user to percieve it as showing up instantly.
Currently I'm doing this using a canvas and textblocks, one textblock per 80-character line, with multiple runs to a textblock to handle the different colors. There can often be 20+ runs to a line.
In a 45x80 grid of text, this gets a framerate of about 12 in Silverlight, which is sluggish. Oddly enough, it's even slower in WPF... I can only speculate that this has something to do with my dev machine's lousy video card. Maybe it's trying to hardware accelerate and actually slowing down because of it?
I created a program that watches about 10 folders for different excel files. Using the filesystemwatcher with filters in place and correct error handling, it still misses some files once in a while (that's ok I guess).Anyways, I'm writing this because I added another form that has a filesystem watcher, when I tried to load the form I got a stack overflow error - an error that isn't debugged (a greenish line that you can only stop debugging).Am I going about this a wrong way? Is filesystemwatcher unreliable with large amounts of processing?
Couldn't think of a better title.(Background on the problem/me)Okay, so, first question/post here, so hi. Now that that's done with, the information pertinent to my problem. I'm fairly new to VB (and programming as well, aside from screwing around with C++ and learning assembler(well, attempting is the better word) god knows how many years ago), and have only seriously been programming for about under half a year, and my skill level is about at that stage. Only been using VB.Net, nothing older. Depending on the time of day and if I'm home or at school, I fluxuate between VB express and Vis Studio 08. Umm, this program I'm having trouble with was on a test that I took yesterday (took the problem home with me cause I really wanted to figure out what was wrong with it).
The stipulations of the test were: No For->Each Loops No using Built-In Sorting or Searching Functions
I have a user interface app that allows you to drag/drop in files - and it turns them into HttpListener request POST's to another EXECUTABLE that I have running - let's call it BACKEND1. BACKEND1 will be running on a SERVER when this gets ready for production use. Lots and lots of users running the UI - all dragging in files - all getting POST'ed to BACKEND1 - with the file included as a memory stream in the POST. BACKEND1 responds to the UI with a "sequential" file number assigned to the file centrally on the server.
On the SERVER is also running another EXECUTABLE - let's call it BACKEND2. It's job is to work with the DRAG'd in files. Both BACKEND1 and BACKEND2 sit on the same SERVER. At the moment I am writing the memory stream to DISK in BACKEND1 - and POSTing the FILENAME to BACKEND2 for it to work on. Would it be better to not write the file in BACKEND1 but instead include it in the POST to BACKEND2 - where it can be written to DISK? Is having LARGE HTTP REQUEST's a burden when sent between two EXECUTABLES on the same machine?
My primary goal is to have a really available BACKEND1 talking the the UI instances running out in the world. I'm concerned that doing the file i/o in BACKEND1 is "expensive" - and I don't care about the performance of BACKEND2 as much. just looking for opinions. I guess ultimately I'll have to bench mark this myself in a production environment to get a real answer.
I have a user interface app that allows you to drag/drop in files - and it turns them into HttpListener request POST's to another EXECUTABLE that I have running - let's call it BACKEND1.BACKEND1 will be running on a SERVER when this gets ready for production use.Lots and lots of users running the UI - all dragging in files - all getting POST'ed to BACKEND1 - with the file included as a memory stream in the POST. BACKEND1 responds to the UI with a "sequential" file number assigned to the file centrally on the server.
On the SERVER is also running another EXECUTABLE - let's call it BACKEND2. It's job is to work with the DRAG'd in files.Both BACKEND1 and BACKEND2 sit on the same SERVER. At the moment I am writing the memory stream to DISK in BACKEND1 - and POSTing the FILENAME to BACKEND2 for it to work on.Would it be better to not write the file in BACKEND1 but instead include it in the POST to BACKEND2 - where it can be written to DISK?
Is having LARGE HTTP REQUEST's a burden when sent between two EXECUTABLES on the same machine?My primary goal is to have a really available BACKEND1 talking the the UI instances running out in the world. I'm concerned that doing the file i/o in BACKEND1 is "expensive" - and I don't care about the performance of BACKEND2 as much.
I am facing an amazing problem in my application. On my Developement server when I am running my application, then there is one function where I am trying to put whole stringbuilder contents in a string object. This Function is used for paging the reports.For that I am using a method of Stringbuilder.tostring() and when I chek the value of this statement (stringbuilder.tostring()) it shows no such interface is supported, which later on throws exception of system is out of memory. Here i am using .net 2.0 framework and sql server 2005 database.But when I run the application on stagging server then there is no such exception thrown.[code]
For example, to calculate this:(22 ^ 29) Mod 57Supposedly there is a way to get around overflows and do these calculations, but I can't find much online. Now, the person who said this was doing VBA in Excel and said that there was about 4 lines of code which could handle the overflow so it wouldn't happen.
I have the following code that connects to a server and expects data from it.Is it normal that working memory of my program increases every time I receive something from the server I connect to?The larger I make the network buffer, the more increase that is added to my working memory (in task manager)The data I receive is about 50 bytes. So I minimized that size of the buffer to 128.
Some of my users are collecting data files this size and wish to load it in to my tool. Is there a way to load large files of string data w/o getting a OutOfMemoryException? I have tried many different storage methods (Array, List, etc) but always without success.Before anyone asks 'why load it all at once', let me say the nature of my app is speed critical and it must be so.
Ok so i have an jpg image of around 8.10mbs at around 13424 x 11344 pixels, i need to load the whole image in to a scrolling picturebox, is they anyway to do this without using so much memory for example like what google maps does n cuts the image down and some how stitches it all back to gether?
I have a problem with a memory leak in a very large ASP.NET application. After about 24 hours of usage (sometimes a lot more) an OutOfMemory exception is thrown. Therefore I am trying to understand how managed memory works in .NET. When does an ASP.NET application throw an OutOfMemory exception? The server has a lot of RAM and there is always plenty of memory left when an OutOfMemory exception is thrown. I understand the difference between virtual memory and physical memory. How much memory does the W3WP process have to consume before an OutOfMemory exception is thrown? Is there a setting somewhere e.g. in the Machine.Config file? The ASP.NET process is never recycled.
I need to improve memory performance on my application and I could see that I have problems with memory fragmentation.I've read an interesting article on large objects from Andrew Hunter of Red Gate, and one of the solutions he recommends isow do I implement his suggestion in my code?My program has a very complex form (with an object that leaves residual memory every time it opens. I found a complex list that may be the culprit, and I'd like to implement his suggestion to see if it fixes the issue.
I'm loading an image, then getting size and such info from it, and loading the image into a picture box. Now it works and all, but I noticed it uses a large amount of memory. I'm doing: img = Image.FromFile(file)Where I load the image, 'file' being just a string containing path to an image on the hd. After that, I get the size and frame count (if any), and load it into a picture box: The image I've been testing with, is a 6600x5100 3.5MB JPEG, which can be downloaded from url...Watching task manager, when the app loads the image, it takes up about 104MB of memory. I've tried doing img.Dispose after the above line, but it brings up an error. If I put a button on the form with img.Dispose in it, and click it after app has loaded, it drops down to about 6MB of memory used. Loading the image directly into the picture box using .Load or .LoadAsync, after disposing img beforehand, still ends up with a huge amount of memory used.
So I'm wondering if theres a way to check if the picture box has fully loaded the img, so then I can Dispose the img and free up the memory. Or maybe a better way all together of loading an image into a picture box, and/or getting size & frame count, without having to use up that amount of memory before freeing.
I am retrieving a large BLOB and keep getting an Out Of Memory Exception. It occurs when I use a SqlDataReader to either check if the column is NULL, or if I try to read it and assign it to a Byte type of parameter, or if I use Response.BinaryWrite.
When reading from a network stream using the following [code] I tend to get the data in two or more chunks, unless I specify a delay after each buffer read (which is hardly desirable). It is usually a problem with 4k+ buffers written (in one large chunk and 'flushed') to the networkstream and read in smaller chunks (of say 1k).This is seriously problematic for the rest of the code, since the information is broken up in bits and not properly handled.Is there a way to ensure that all data available is 'captured' without using unrealisticly large buffers or using delays?
So, I have this code that grabs a bunch of data from the database, and performs a bunch of calculations on that data. However, this is not what is causing the halt. The halt is coming in when I take all that "final" data that has been prepared, and go to write that into a text file.
Each line in the text file is being created from data that has had calculations performed on it. Each line "corresponds" to a record from the database. A user will have to perform this file export usually on about 30-40 thousand records at a time. Now, even though I am using a StringBuilder, this still halts because of the pure mass amount of records.
What I do currently in the code is create one StringBuilder, append all the data (after calcs) to it, convert it to a String, add it to a List(Of String) to be returned from the Function, and then clear the StringBuilder, and do the same for the next row, and I do this until each row has been created and appended, converted, and then inserted into the List.
Then, another class receives this List(Of String) and takes each String and eventually appends it to a file.
Does anyone have an idea of how to improve this code and make it so it doesn't halt the server every time? The reason it wasn't found out until now is because when I tested it locally I was testing it on only 1-50 records at a time.
I need some guidance on a little program I'm making (one that I thought would take a couple of hours from start to finish, it's been days now ...) that determines all of the proper divisors of any given number up to the maximum value of a UInt32.
I seem to be having some issues with my code here. The sub "CheckSettings" is actually run off of a system timer, so that every 5 seconds it calls CheckSettings (actually, i set it to every 1 second for debugging, so i can see the leak a little more clearly by monitoring the process). CheckSettings basically checks to ensure there isn't already an instance of the thread running, and if there is, it checks to ensure that the instance has completed running ... once it has the go ahead, it creates a new thread and runs again.
code:
This code is creating a very large memory leak. After a few minutes of running, my application goes from a memory footprint of about 45,000 K to about 1,000,000 K. Quite the leak. I read some articles on the managed versus unmanaged memory etc, and I was careful to free the objects etc.
The company require an application that displays coffee usage information for the managers of CostaB Coffee. Sample data has been provided for last years (2011) monthly usage amounts in kilos and is as follows: 400.5, 450, 475.5, 336.5, 457, 325, 220.5, 276, 300, 320.5, 400.5 and 415.
Requirements: The main form will allow the user to enter and store monthly amounts in an appropriate data structure. It will enable the users to calculate the total usage for the 12 month period and average monthly usage. A further requirement for the application is for the users to be able to display the monthly totals in ascending (smallest to highest) and descending (highest to lowest) order of the monthly usage amounts. Utilise appropriate formatting for input/output and include appropriate error trapping and data validation in the application.
The loop will run about 3 times then overflow. I guess no matter what u do xor will result in Byte,Integer or Long. CDec(Key) will not work I need a way to bypass this so i can Xor large numbers.
How can I calculate 2^ 3333 in vb.net it calculates up to 2^ 60 it says infinite for this calculation. Is tehre any way to work with really large numbers?
I am using Virtual Basic with ASP.NET in Visual Studio 2008. I am using Webmethod to communicate from/to the server with the client. Data can be queried and returned in small amount, but when I try get a relatively large amount of data, it's returning me a Server 500 error. My data is 226561 character long, so it's not that large, but it's relatively larger than my other working testing set, which are about 10k character long.
JavaScript: PageMethods.my_func("context", success); function success(result, userContext, methodName) {
I use the following Rijndael code to do encryption without fail for many times. But why it can not encrypt an ISO file with 4.2 GB? In fact my computer has 16GB memory and it should not be a memory problem. I use Windows 7 Ultimate. The code is compiled as winform (.Net 4) using Visual Studio 2010 (a VB.NET project).
I have checked that the ISO file is OK and can be mounted as virtual drive and even can be burnt to DVD rom. So it is not the ISO file problem.My question: Why the following code cannot encrypt an ISO file with size of 4.2GB? Is that caused by the limitation of Windows/.NET 4 implementation?
I am uploading a file from a C# application to an ASP.Net website, both written by me so I have access to the code.But, it is working for a text file (1KB) but not for a MOV file (77MB).
In both cases, I am using UploadProgressChanged to get the progress. The TXT file goes to 100% while the MOV file goes only till 50%. When done, I only find the TXT file saved on the server but not the MOV file.
