Some of my users are collecting data files this size and wish to load it in to my tool. Is there a way to load large files of string data w/o getting a OutOfMemoryException? I have tried many different storage methods (Array, List, etc) but always without success.Before anyone asks 'why load it all at once', let me say the nature of my app is speed critical and it must be so.
View 3 RepliesI am creating a VB application in Visual Studio 2010 that installs some SQL stored procedures. These stored procedures are encrypted otherwise I would just supply my clients with .sql files to create the stored procedures.
The install script that creates the stored procedure is 1500 lines in length and has been formatted in the way VS 2010 requires multi-line literals to be in:
"First Line" & _
"Second Line" & _
"etc..."
[Code]....
I am working on a program for a customer and am working in VB2010 in VB language. I have two rather large files that contain data. File #1 is over 300,000 lines of data. File #2 is over 100,000 lines. The files are CSV text files. On a User Form, there are two values, "StartTime" and "StopTime". This is the process I am doing:
1. Read both files into ArrayLists
2. Get item #1 of ArrayList #1 (we'll call this "A")
3. Get item #1 of ArrayList #2 (we'll call this "B")
[code]....
I have a series of large files (4GB) that when I delete takes a very long time. This only occurs on a USB drive that I have attached to my PC. If I do the same thing on the internal drive the files delete quickly. It is as if the files are being overwritten before being deleted. I am looking for some guidance or other approaches.
View 5 RepliesI am trying to develop an application that will let me do the following
1) Be able to search to large binary files for a specific byte array (header) and to continue searching for another byte array (footer). Then i want it to copy the section of the binary file between these 2 locations to a specified file. Then the program is to continue searching for any other header/footers An example would be search a large file and carving out the jpgs it finds etc. This routine would have to be able to handle raw images of 20gb or more if needed.
so I want to make a button in one of my programs that writes text to a file. I know how to do that part. I use:
[Code]...
I recently wrote a program that collects information from the network can displays it in a DGV for the user to see. I tested this on networks where I would end up with several thousand items brought back and all worked fine, very little memory increase and once the search had finished the memory usage went back to pretty much what it was before the search, so I assume I have no major memory leaks that I should be concerned about (I'm careful to always dispose of things and clear large variables where possible).
However, a new company has now started using my application and they have several hundred thousand items that my program will find. They reported that they leave the program running its search for a few hours and most of the time it crashes with an Out Of Memory exception. Now I did have my app do some caching of certain information to avoid querying the network more than was necessary so I thought well maybe this is what is causing it but I added an option to turn that off and they say it still happens. So the only thing I can see it being is simply the amount of data that the program is finding and having to store in memory.once my application has gathered information about the items it stores them in a List(Of MyItemClass) and then when the search is complete the items are added to the DGV. So I'm thinking maybe its just the size of this List(Of MyItemClass) that is simply getting too large. Bear in mind that each instance of MyItemClass in this List can have quite a lot of information in it as it has several properties that are List(Of String) that may contain a few thousand strings in some scenarios.How do you deal with such large amounts of data without running out of memory?I'm assuming the only option I have got is to 'page' some of the data to a temporary file on disk once I get over a certain number of items in the List(Of MyItemClass)?
When reading from a network stream using the following [code] I tend to get the data in two or more chunks, unless I specify a delay after each buffer read (which is hardly desirable). It is usually a problem with 4k+ buffers written (in one large chunk and 'flushed') to the networkstream and read in smaller chunks (of say 1k).This is seriously problematic for the rest of the code, since the information is broken up in bits and not properly handled.Is there a way to ensure that all data available is 'captured' without using unrealisticly large buffers or using delays?
View 4 RepliesCurrently we have a large DataTable (~152k rows) and are doing a for each over this to find a sub set of distinct entries (~124K rows). This is currently taking about 14 minutes to run which is just far too long.
As we are stuck in .NET 2.0 as our reporting won't work with VS 2008+ I can't use linq, though I don't know if this will be any faster in fairness.
Is there a better way to find the distinct lines (invoice numbers in this case) other than this for each loop?[code]....
I wrote a program to download and play AVI files the files are large in size and long time wise. I have the files playing as they are being downloaded but i can't see how long the file is or seek reliably. i was reading that the AVI headers are in the last 512kb of the AVI file *why would't they put it at the start* lol so my question is is there a way to download the last 512Kb of the file i'm downloading. i have the Bytes Read and Length of the file while its downlaoding i'm just not sure where to go from there, or how to do it atleast. If i had the length in time of the Avi file i could set the trackbar to be able to seek properly. or if someone has an idea how i could get the time of the video by using fps and some math i could prob do it that way too but idk how i would tell how i can find how many kb are in the fps i'm sure it changes so think that way is't going to be reliable. how to get the last 512kb would be the best option not sure if it can be done even.
View 4 RepliesHere's what i've got so far, i've written all of the data that i'm going to put into my config file, however i'm having trouble reading the specific string values that I need from that ini file. The parts of the ini lines I want to get are the values in between the I've tried using streamreader a couple things but i'm having trouble figuring that out. I need to parse each line of the ini file to get that value in between the single quotations, so that I can set the value for my richtextbox background and text color as well as a few other things that i've set up in my notepad including tab spacing and so on. lots of these different values need to be converted from a string value to integers, and boolean values as well. (4 of them, the rest of them don't have to be changed because they are already string values when my application reads them to determine it's application load settings.
EDIT: I just noticed in the preview that this is from the version of the app where I had streamwriter write "______ value=" & Something to a file. lol some of the (')s are still in this version, so please excuse that little part. I had them in, but took them out so that I could trim data to get a value easier, but I want to use (')s hopefully when all is said and done.For example, just to restate what I want - in the ini file it will read [setting name] value='VALUE'I want to be able to "get" the string value of "VALUE" within the ini file between the 2 single quotations '
When you need to do random access to the lines of a text file, the regular solution is to put those lines in a list and access them in that list.But doing this if the text file is large creates a large memory overhead and may cause some memory stress on the application.This class maps the file and provide the ability to read any particular line in the file at a random position, without the need to read or put in memory any other line but the desired line. The only memory overhead involve is the 1024 bytes buffer of the base stream.Even if the file mapping can be a process that consume a noticable amount of time if the file is very large ( ~ 5 seconds for each 100,000,000 caracteres) , once the mapping is done the access to a particular line becomes instantaneous regardless of the size of the file. (The file mapping can always be done at a convenient time (ex when the process load) and can be done on a worker thread The class is base on a StreamReader to support the differents Text Encodings Example uf usage ( As I say, if the file is very large, dont do it like that, but declare the class at a convenient time)
Imports Reader = System.IO.MyStreams
Public Class Form1
Private Sub Button1_Click(ByVal sender As System.Object, ByVal e As System.EventArgs) Handles Button1.Click
[code]....
How do you recommend I copy large files over the LAN (>20GB)Currently I am using File.Copy.Another question I have is how can I implement this class to be self contained, meaning that it would report progress to the object that called it? or do I have to define a delegate in the calling object?
Code:
Public Class cCopier
Private Delegate Function CopyProgressRoutine(ByVal totalFileSize As Int64, ByVal totalBytesTransferred As Int64, ByVal streamSize As Int64, ByVal streamBytesTransferred As Int64, ByVal dwStreamNumber As Int32, ByVal dwCallbackReason As Int32, ByVal hSourceFile As Int32, ByVal hDestinationFile As Int32, ByVal lpData As Int32) As Int32
[code].....
The underlying connection was closed: An unexpected error occurred on a receive.If I look on te FTP server I can see that the file has been uploaded entirely which led me to believe that the control channel was timing out. I have tried modifying the timeout on the FTP server as well as the ReadWriteTimeout in my code to no avail.Here is my code...
Dim ftprequest As System.Net.FtpWebRequest = DirectCast(System.Net.WebRequest.Create(destination), System.Net.FtpWebRequest)
ftprequest.Credentials = New System.Net.NetworkCredential(username, password)
[code].....
I am opening a large PDF file using the following
Process.Start(Application.StartupPath & "PREVI.pdf")
I have two questions:
1) Is there a way to go directly to a page number?
2) Can I write a code to have the table of contents show up on the left side of the document so users can click on to go to different sections of the document, not sure if this needs to be done while creating the PDF or if we can do it by code.
If you were to upload a project to any site but are concerned about the large file size, what are the only files you need to zip up for your project to work for either VB 2008 or VB 2010? So far I know you don't need the exe's, the pdbs, and xml files.
View 5 RepliesWhich is the most performant way to read a large csv file in .NET? Using FileStream? or another class
View 4 RepliesDoes anyone know what the best way to split up large files in VB.NET? These files can be in excess of 10GB. I have found ways of doing it by googling all day! most of the solutions I have found almost work. But what I really want to know is what is the most efficient way to do this?
View 1 RepliesI have trying to upload some large files using FTP. Here is the code I am using below.
Private Sub UploadFTP()
' Open file.txt with the Using statement.
Using r As StreamReader = New StreamReader("list.txt")[code]....
I basically loop the above to upload all the files in a specific directory. This works fine with small text files,but when I try uploading a large 30+- mb file it gives me an error.
Quote: The underlying connection was closed: An unexpected error occurred on a receive.
On
cls_stream.Close()
I understand this is caused by .Net not communicating with the server properly? I tried searching all morning to only get confused by all the different methods.
Say I want to remove all lines that contain "hello". One way to do it is to read the whole files into memory and then operate. Often my files are big, 500MB. If I load them I got no memory. Will upgrading to a 64 bit computer helps? I have a big page file. Also I wonder how in the earth windows 7 and xp 32 bits can have very large page files anyway.
View 9 RepliesI have created an vb.net windows application. In the application i have 2 xml files (e.g., Pre.xml & Post.xml) & the 2 files are same structure.
My requirement is compare the pre & post xml files and get the differences between xml files. Like what are all the newly added nodes, removed nodes and modified nodes as well.
I tried with XmlDiffPatch.Dll, it is working for small size xml files, but in my case xml file size should be upto 100+ mb.
I am working on a application that stores videos, images and files. I would like to give the user the possibility to encrypt them. I am tying to use AES to encrypt them but the problem is that the decryption takes to long.I was hoping that the loading time from the hard disk of the encrypted file was far longer than the process of decryption of an image for example but i was wrong
So if i try to 1) load the image 2)decrypt it and 3)load it in a image control it takes 15 times more time that loading the original file not encrypted
I am using this code to write files
Try
Dim ObjWS As New localhost.Service
'ObjWS.AssigningDataToTemplate(1550, arrValues)
[Code]....
Here i got proble to send a large size of file like 500 mb file etc
Just a doubt: if I have large amount of image files for my app, is it ok to store them in the resource file ?
Will those files be saved in a single resource file ? Will that cause any problem to the performance of my app ? When I try to access a image file from the resource using code, will it load that file only into the memory or the entire image contents from the resource file ?
I am writing a small tool that identifies digital scielence in non compressed PCM WAV files. Its purpose will be to scan files approximately 1 hour long, containing mostly scilence and spltting the periods of non scilence into files and discarding any digital silence which is encoded as null bytes.I am wondering what the most efficient way of scanning the files is. I am able to derive the number of bytes for each second from the file header (in my case 16000) so am considering reading into a buffer of this size then scanning byte for byte, if I identify a non-null byte I will start siphoning the data off to another file.My issue is with scanning large expanses of digital silence which could be many megabytes of data which would require examining each byte to determine wether or not its digital silence. Surely there is a more efficient way of skipping them as I am not interested in them.
View 7 RepliesMy problem is I have very large text files (approx 2GBs+).They have records in them based in one per line.Each line is not the same length and the data can be different lengths all the time.I am currently reading the file line by line, then splitting the data by common characters in the records. To process the full file it currently takes 3hours. This is way too slow for its purpose.
View 7 RepliesSo I can encrypt small files using Des Encryption, but whenever I try encrypting large files, The program's memory usage skyrockets! Would anybody be able to get me some code that will let me encrypt large files (It doesn't have to be Des, but I need to be able to support any length password).
View 1 RepliesI have two large text files (about 1.8 GB each encoded in UTF-8). And I want to check if the content of both files are 100% same. If there is single character difference then it is considered not same, i.e. both files are different. How do I do that? So my question is: How to check if 2 large text files content are 100% same?
[Code]...
I have a large file (2.7GB). I need to split it into smaller files. How to split a large file into smaller files using VB.NET 2003? Cannot use LINQ and the resources (cpu and memory) on operating environment are very limited (it is a shared hosting environment).
View 1 Replies[Code] each file is about 30 kilobytes in size, and they contain raw hardware statistical information in a comma-deliminted format. I want to do the following with these files as my end-result.
NEWFILE.CVS contains the contents in this format:
metric_group_001 metric_group_002 metric_group_003 metric_group_004 etc.
I don't want APPEND, I want to concatenate the contents of these files into one large master file. I am able to do this MANUALLY, but I need to have a DYNAMIC method of doing this because of the number of files will change depending on the test we are doing with the hardware. This is the code I am using to do this process manually and it does work, but I need a DYNAMIC method in place. [Code]