Reading Large Csv Files
Sep 17, 2010Which is the most performant way to read a large csv file in .NET? Using FileStream? or another class
View 4 Replies
ADVERTISEMENT
Reading Large Files Efficiently
Sep 12, 2010I am writing a small tool that identifies digital scielence in non compressed PCM WAV files. Its purpose will be to scan files approximately 1 hour long, containing mostly scilence and spltting the periods of non scilence into files and discarding any digital silence which is encoded as null bytes.I am wondering what the most efficient way of scanning the files is. I am able to derive the number of bytes for each second from the file header (in my case 16000) so am considering reading into a buffer of this size then scanning byte for byte, if I identify a non-null byte I will start siphoning the data off to another file.My issue is with scanning large expanses of digital silence which could be many megabytes of data which would require examining each byte to determine wether or not its digital silence. Surely there is a more efficient way of skipping them as I am not interested in them.
View 7 RepliesReading Large Text Files
Jan 30, 2012My problem is I have very large text files (approx 2GBs+).They have records in them based in one per line.Each line is not the same length and the data can be different lengths all the time.I am currently reading the file line by line, then splitting the data by common characters in the records. To process the full file it currently takes 3hours. This is way too slow for its purpose.
View 7 RepliesReading Large Excel Files Or Sheets?
Feb 10, 2010I'm tryng to read an excel sheet that has more than 255 cell. I'm using the following code:
Dim MyConnection As System.Data.OleDb.OleDbConnection
Dim MyDataSet As New System.Data.DataSet
Dim connectionString As String = ConfigurationManager.ConnectionStrings("PCOConnectionString2").ConnectionString
[Code].....
this code throws me an error that indicates me that there are too many rows to read. I found, seaching through forums, that it can't be more that 255 cells on the sheet i'm reading.
How can i read this sheet that has over 500 cells?
File I/O And Registry :: Reading Files From A Large Directory?
Jul 15, 2009I wrote a cleanup program to go through some directories and delete files based on if the creation date is older than say 6 months. It works fine with some of the directories I have that contain around a few thousand small files. However, there is one directory (that contains small backup files from another program) that is loaded with over 300,000 files and it locks up on me as soon as I read in the first file in that directory.I am convinced it is the directory has too many files in it to open it. The server that the directory is on is slow. It takes a half hour to open the directory while on the server itself. I know it will take forever to delete the amount of files I want to delete, but I don't understand why it gets stuck and hangs there with no error message.Here is where I get stuck. Listbox1 is the directory I'm attempting to access
For Each selectFile In My.Computer.FileSystem.GetFiles(ListBox1.Items.Item(Count), FileIO.SearchOption.SearchTopLevelOnly, "*.*")
compFile = Path.GetFileName(selectFile)
[code].....
Reading And Parsing Large Delimited Text Files?
Nov 23, 2011I'm busy with an applicaton which reads space delimited log files ranging from 5mb to 1gb+ in size, then stores this information to a MySQL database for later use when printing reports based upon the information contained in the files. The methods I've tried / found work but are very slow.
or is there a better way to handle very large text files?
I've tried using textfieldparser as follows:
Using parser As New TextFieldParser("C:logfiles estfile.txt")
parser.TextFieldType = FieldType.Delimited
parser.CommentTokens = New String() {"#"}
[Code].....
File I/O And Registry :: AVI Header - Program To Download And Play AVI Files The Files Are Large In Size And Long Time Wise
Dec 11, 2011I wrote a program to download and play AVI files the files are large in size and long time wise. I have the files playing as they are being downloaded but i can't see how long the file is or seek reliably. i was reading that the AVI headers are in the last 512kb of the AVI file *why would't they put it at the start* lol so my question is is there a way to download the last 512Kb of the file i'm downloading. i have the Bytes Read and Length of the file while its downlaoding i'm just not sure where to go from there, or how to do it atleast. If i had the length in time of the Avi file i could set the trackbar to be able to seek properly. or if someone has an idea how i could get the time of the video by using fps and some math i could prob do it that way too but idk how i would tell how i can find how many kb are in the fps i'm sure it changes so think that way is't going to be reliable. how to get the last 512kb would be the best option not sure if it can be done even.
View 4 RepliesReading A Very Large Text File?
Jan 30, 2012My problem is I have very large text files (approx 2GBs+).They have records in them based in one per line.Each line is not the same length and the data can be different lengths all the time.I am currently reading the file line by line, then splitting the data by common characters in the records. To process the full file it currently takes 3hours. This is way too slow for its purpose.
View 4 RepliesReading Large Text File?
Jul 6, 2011I've a problem reading text file using StreamReader. The file have between 500 000 and 1 000 000 lines.When I try to read it in a cycle, I get an error. That's why I've tried the StreamReader.ReadToEnd method. It worked fine. I've get the entire contents of the file in one string. So far everything is okay, but I've a small problem searching this huge string. I have to reformat the string to my desired format. I'll try to be more specific: The format of the input file is as follows:
50471100 8 2 6 5 0<LF><CR>
00000016 365442 12231<LF><CR>
00000026 112166 31133<LF><CR>
<end>
[Code]...
[2008] Reading A Very Large Text File
Feb 1, 2009The following code suppose to:
1. Read line-by-line a txt file with more than 500,000 lines, (each line 521 characters long)
2. extract an ID No from the line
3. query from a database for LCCIStatus
4. concatenate the value of LCCIStatus to the line
5. write the line to sample.txt
My problem is, this code works perfectly with the test file of 8000 lines but fail with the actual files which have over 500,000 lines. FYI, the test file contains data which I cut and paste from the actual file.
[Code]...
Reading Large File Stream In .net (How Can I Increase Buffer Size)?
Mar 3, 2009i need to read a large file stream binary my code is Dim sr As New IO.FileStream(srcFile,
[Code]..
Serial Port Timing - Reading Large Flash Contents
Feb 10, 2011I am having an issue with serial port timing when using it to read large flash content. If I slowly step through the code the program works as expected. If I let it runs by itself it only shows part of the result. The problem is not related to displaying the data on the form but mainly the "Serialport.BytesToRead".
Here is the code
Private Sub MemoryReport_Click(ByVal sender As System.Object, ByVal e As System.EventArgs) Handles MemoryReport.Click
Me.BackgroundWorker1.RunWorkerAsync()
'Memory_Report()
'SerialPort.DiscardInBuffer()
[Code] .....
Odbc - Reading A Large File (>5MB) From A BLOB (container) Field From Filemaker?
May 4, 2011I'm trying to read a binary file from a FileMaker 11 container field using Filemaker's own ODBC driver. I was able to write files to the database and this works fine. retrieving them manually works fine and the files look OK and are not corupted. However when retreiving them using VB.NET, and if the file size is approx > 5MB, I get the following "uncatchable" error (yes thats right, I cant "Try Catch End Try", it just crashes):
System.AccessViolationException was unhandled
Attempted to read or write protected memory. This is often an indication that other memory is corrupt.
[code].....
Copying Large Files Over LAN?
Jan 29, 2009How do you recommend I copy large files over the LAN (>20GB)Currently I am using File.Copy.Another question I have is how can I implement this class to be self contained, meaning that it would report progress to the object that called it? or do I have to define a delegate in the calling object?
Code:
Public Class cCopier
Private Delegate Function CopyProgressRoutine(ByVal totalFileSize As Int64, ByVal totalBytesTransferred As Int64, ByVal streamSize As Int64, ByVal streamBytesTransferred As Int64, ByVal dwStreamNumber As Int32, ByVal dwCallbackReason As Int32, ByVal hSourceFile As Int32, ByVal hDestinationFile As Int32, ByVal lpData As Int32) As Int32
[code].....
FtpWebRequest And Large Files?
Jul 24, 2010The underlying connection was closed: An unexpected error occurred on a receive.If I look on te FTP server I can see that the file has been uploaded entirely which led me to believe that the control channel was timing out. I have tried modifying the timeout on the FTP server as well as the ReadWriteTimeout in my code to no avail.Here is my code...
Dim ftprequest As System.Net.FtpWebRequest = DirectCast(System.Net.WebRequest.Create(destination), System.Net.FtpWebRequest)
ftprequest.Credentials = New System.Net.NetworkCredential(username, password)
[code].....
How To Open Large PDF Files
May 13, 2009I am opening a large PDF file using the following
Process.Start(Application.StartupPath & "PREVI.pdf")
I have two questions:
1) Is there a way to go directly to a page number?
2) Can I write a code to have the table of contents show up on the left side of the document so users can click on to go to different sections of the document, not sure if this needs to be done while creating the PDF or if we can do it by code.
Large Files Are Not Uploaded
Dec 20, 2011If you were to upload a project to any site but are concerned about the large file size, what are the only files you need to zip up for your project to work for either VB 2008 or VB 2010? So far I know you don't need the exe's, the pdbs, and xml files.
View 5 RepliesSplitting Large Files With .NET?
Apr 27, 2012Does anyone know what the best way to split up large files in VB.NET? These files can be in excess of 10GB. I have found ways of doing it by googling all day! most of the solutions I have found almost work. But what I really want to know is what is the most efficient way to do this?
View 1 RepliesUploading Large Files Using FTP?
Mar 12, 2012I have trying to upload some large files using FTP. Here is the code I am using below.
Private Sub UploadFTP()
' Open file.txt with the Using statement.
Using r As StreamReader = New StreamReader("list.txt")[code]....
I basically loop the above to upload all the files in a specific directory. This works fine with small text files,but when I try uploading a large 30+- mb file it gives me an error.
Quote: The underlying connection was closed: An unexpected error occurred on a receive.
On
cls_stream.Close()
I understand this is caused by .Net not communicating with the server properly? I tried searching all morning to only get confused by all the different methods.
Compare 2 Large Size Xml Files?
Jan 29, 2010I have created an vb.net windows application. In the application i have 2 xml files (e.g., Pre.xml & Post.xml) & the 2 files are same structure.
My requirement is compare the pre & post xml files and get the differences between xml files. Like what are all the newly added nodes, removed nodes and modified nodes as well.
I tried with XmlDiffPatch.Dll, it is working for small size xml files, but in my case xml file size should be upto 100+ mb.
Encrypt Large Files Quickly?
Jun 24, 2009I am working on a application that stores videos, images and files. I would like to give the user the possibility to encrypt them. I am tying to use AES to encrypt them but the problem is that the decryption takes to long.I was hoping that the loading time from the hard disk of the encrypted file was far longer than the process of decryption of an image for example but i was wrong
So if i try to 1) load the image 2)decrypt it and 3)load it in a image control it takes 15 times more time that loading the original file not encrypted
How To Write Large Size Of Files
Apr 13, 2009I am using this code to write files
Try
Dim ObjWS As New localhost.Service
'ObjWS.AssigningDataToTemplate(1550, arrValues)
[Code]....
Here i got proble to send a large size of file like 500 mb file etc
Large Amount Of Image Files For App?
Jun 19, 2010Just a doubt: if I have large amount of image files for my app, is it ok to store them in the resource file ?
Will those files be saved in a single resource file ? Will that cause any problem to the performance of my app ? When I try to access a image file from the resource using code, will it load that file only into the memory or the entire image contents from the resource file ?
Large Files For Data Comparison?
Sep 21, 2010I am working on a program for a customer and am working in VB2010 in VB language. I have two rather large files that contain data. File #1 is over 300,000 lines of data. File #2 is over 100,000 lines. The files are CSV text files. On a User Form, there are two values, "StartTime" and "StopTime". This is the process I am doing:
1. Read both files into ArrayLists
2. Get item #1 of ArrayList #1 (we'll call this "A")
3. Get item #1 of ArrayList #2 (we'll call this "B")
[code]....
VS 2008 - How To Encrypt Large Files
Oct 16, 2009So I can encrypt small files using Des Encryption, but whenever I try encrypting large files, The program's memory usage skyrockets! Would anybody be able to get me some code that will let me encrypt large files (It doesn't have to be Des, but I need to be able to support any length password).
View 1 RepliesVS 2010 Deleting Large Files?
Jul 6, 2010I have a series of large files (4GB) that when I delete takes a very long time. This only occurs on a USB drive that I have attached to my PC. If I do the same thing on the internal drive the files delete quickly. It is as if the files are being overwritten before being deleted. I am looking for some guidance or other approaches.
View 5 Replies.net - Check If 2 Large Text Files Content Are 100% Same?
Jun 29, 2011I have two large text files (about 1.8 GB each encoded in UTF-8). And I want to check if the content of both files are 100% same. If there is single character difference then it is considered not same, i.e. both files are different. How do I do that? So my question is: How to check if 2 large text files content are 100% same?
[Code]...
.net - Split A Large File Into Smaller Files?
Oct 8, 2011I have a large file (2.7GB). I need to split it into smaller files. How to split a large file into smaller files using VB.NET 2003? Cannot use LINQ and the resources (cpu and memory) on operating environment are very limited (it is a shared hosting environment).
View 1 RepliesConcatenate Multiple Files Into One Large File
Aug 16, 2009[Code] each file is about 30 kilobytes in size, and they contain raw hardware statistical information in a comma-deliminted format. I want to do the following with these files as my end-result.
NEWFILE.CVS contains the contents in this format:
metric_group_001 metric_group_002 metric_group_003 metric_group_004 etc.
I don't want APPEND, I want to concatenate the contents of these files into one large master file. I am able to do this MANUALLY, but I need to have a DYNAMIC method of doing this because of the number of files will change depending on the test we are doing with the hardware. This is the code I am using to do this process manually and it does work, but I need a DYNAMIC method in place. [Code]
File I/O And Registry :: Zip Large Number Of Files?
Aug 28, 2009I am using zipforge.net to archive files in a small backup program I am creating. Everything works fine however if I try to archive a large number of files the process of zipping them gets slower the further along in the job it gets.
Example: I have a directory that contains a little over 100,000 files in it. Each file is only about 200k of text. When I use the zipforge zip class it starts zip'ing very quickly but as it gets about 40% of the way through the directory it starts to slow down... at about 60% its working at a snails pace.
Memory usage for my solution also goes up proportionate to the archive file size as it grows. Currently it takes around 26hours to backup this directory which only contains 4 gigs worth of information.
If I use a program like winzip I can zip the folder in less than an hour. I am thinking there has to be a way to write to the end of the zip file without having to open it every time or without keeping it open. My backup software scans the specified folder and writes each file path and name into a text file. Then it reads the text file line by line and passes the path + file info to the zipforge addfile().