File I/O And Registry :: Zip Large Number Of Files?
Aug 28, 2009
I am using zipforge.net to archive files in a small backup program I am creating. Everything works fine however if I try to archive a large number of files the process of zipping them gets slower the further along in the job it gets.
Example: I have a directory that contains a little over 100,000 files in it. Each file is only about 200k of text. When I use the zipforge zip class it starts zip'ing very quickly but as it gets about 40% of the way through the directory it starts to slow down... at about 60% its working at a snails pace.
Memory usage for my solution also goes up proportionate to the archive file size as it grows. Currently it takes around 26hours to backup this directory which only contains 4 gigs worth of information.
If I use a program like winzip I can zip the folder in less than an hour. I am thinking there has to be a way to write to the end of the zip file without having to open it every time or without keeping it open. My backup software scans the specified folder and writes each file path and name into a text file. Then it reads the text file line by line and passes the path + file info to the zipforge addfile().
I wrote a program to download and play AVI files the files are large in size and long time wise. I have the files playing as they are being downloaded but i can't see how long the file is or seek reliably. i was reading that the AVI headers are in the last 512kb of the AVI file *why would't they put it at the start* lol so my question is is there a way to download the last 512Kb of the file i'm downloading. i have the Bytes Read and Length of the file while its downlaoding i'm just not sure where to go from there, or how to do it atleast. If i had the length in time of the Avi file i could set the trackbar to be able to seek properly. or if someone has an idea how i could get the time of the video by using fps and some math i could prob do it that way too but idk how i would tell how i can find how many kb are in the fps i'm sure it changes so think that way is't going to be reliable. how to get the last 512kb would be the best option not sure if it can be done even.
I wrote a cleanup program to go through some directories and delete files based on if the creation date is older than say 6 months. It works fine with some of the directories I have that contain around a few thousand small files. However, there is one directory (that contains small backup files from another program) that is loaded with over 300,000 files and it locks up on me as soon as I read in the first file in that directory.I am convinced it is the directory has too many files in it to open it. The server that the directory is on is slow. It takes a half hour to open the directory while on the server itself. I know it will take forever to delete the amount of files I want to delete, but I don't understand why it gets stuck and hangs there with no error message.Here is where I get stuck. Listbox1 is the directory I'm attempting to access
For Each selectFile In My.Computer.FileSystem.GetFiles(ListBox1.Items.Item(Count), FileIO.SearchOption.SearchTopLevelOnly, "*.*") compFile = Path.GetFileName(selectFile)
I had tested this code: [URL] It works, but a Word application is opened and each new paragraph added using VB code is seen one by one(with a small delay). If I want to open and write something using VB, what is the fastest way ? I do not want to view the changes made on word document live. I want to save the document. So will oWord.Visible = False makes it fast?
I have to export the data from the database to .txt file, I have the code below that does export it to .txt file but I'm having problems with this code because the data that I'm exporting is about 5 000000 records, it is quite slow and sometimes the pc can't handle it and stops the exporting.
Dim saveFile As New SaveFileDialog If saveFile.ShowDialog = Windows.Forms.DialogResult.OK Then command = New SqlCommand
I have a large file (2.7GB). I need to split it into smaller files. How to split a large file into smaller files using VB.NET 2003? Cannot use LINQ and the resources (cpu and memory) on operating environment are very limited (it is a shared hosting environment).
[Code] each file is about 30 kilobytes in size, and they contain raw hardware statistical information in a comma-deliminted format. I want to do the following with these files as my end-result.
NEWFILE.CVS contains the contents in this format:
metric_group_001 metric_group_002 metric_group_003 metric_group_004 etc.
I don't want APPEND, I want to concatenate the contents of these files into one large master file. I am able to do this MANUALLY, but I need to have a DYNAMIC method of doing this because of the number of files will change depending on the test we are doing with the hardware. This is the code I am using to do this process manually and it does work, but I need a DYNAMIC method in place. [Code]
I am trying to upload 2 large files to a ftp server by uploading as one file to a location. I have it working if the files are small but it crashes with large files.
For i As Integer = 0 To filelist.Count() - 1 Dim fRequest As FtpWebRequest = WebRequest.Create(ftpPath & "/Reports/" & filelist.Item(i)) fRequest.Credentials = New NetworkCredential(username, psswd)
I would like to upload files using the ASP FileUpload control. However, if I try and increase the limit to about 10000 kB, it doesn't work and accepts files only upto 4 MB in size and I get an error saying that the connection to the server was reset. I referred: [URL]..but when I modified the web.config.comments file, nothing happened. Where am I going wrong?
Also, I wanted to know how this would work when I deploy it on the web server. I'm asking because in the code behind, I'm still giving a hard coded value for the path where the file is to be uploaded. How does this work?
I'm using "My.Computer.FileSystem.CopyDirectory(...)" to copy large directories (30-50gb) from one mapped share folder to another. Using "FileInfo.length" to add up all file and subfolder sizes, determines the 'origin directory' size just fine. However, if I use the same method for determining the size of the 'destination directory', mid copy, it returns the full file size, despite the copy not yet being complete.I'm at a loss as to why, as this method works fine in testing, going from a local PC t
Private Function FolderSize(ByVal dir As String) As Long Dim Size As Long = 0 Dim d As New DirectoryInfo(dir)
I am trying to develop an app that will export a large amount of zip files all of which have the same single file in them - simply named 'dw' (with no extension). They will all be in the same directory. I want the file 'dw' to be renamed to the zip file name + .tif. (I will actually eventually need for this to use the magic number to determine if the file is a .tif or a .rtf.) I have been able to get it to cycle through, if the file is NOT renamed. I seem to run into issues when I want the file renamed. It just creates the first .tif file and thats it. I think its actually just cycling through but everything is getting that same name. For the record i am not a programmer by trade, just trying to help with something at work - thus I know the code I have peiced together probably isn't 'by the book'
Dim p As New ProcessStartInfo(ByVal unzipit As String, ByVal zipdir As String, ByVal arg As String, ByVal arg2 As String) Dim di As System.IO.DirectoryInfo di = New System.IO.DirectoryInfo(zipdir) For Each diar In di.GetFiles()
I am wanting to create a large amount of files all copied from a single original file. I then have a list of names in a listbox that each of of these files will rename to. However I am getting an error saying that the file already exists in the save location even though the files are saving to a totally different folder...
code below.
Dim Counter As Integer = listFileNames.Items.Count Do Until Counter = 0 System.IO.File.Move(txtOpen.Text, txtSave.Text)
I am writing a backup program that zips files. Using some sample code I have the following that creates a zip file. Problem is it does not get any subdirectories or files. Where am i going wrong?
Dim azFileNames() As String = Directory.GetFiles(SourceFolder) Dim objCrc32 As New Crc32() Dim zipStream As ZipOutputStream
[CODE]...
Alternativley, there is another component to the zipping dll that I could probably bypass the above altogether. It is: FZ.CreateZip(DestFile, SourceFolder, True, "", "") Where DestFile is (Output stream as system.io.stream). I'm trying to read all the files, directories, sub dir and files, etc from a specific location, let's say 'My Documents' into Destfile in which the FZ.CreateZip zips the file. FZ is FastZip which is from icsharpcode.net. They have c# examples, but that's greek to me. I'm trying to just learn VB!
I've got an app which uses a background worker to copy files. It works fine but some files are quite large and it "appears" as though the app isn't doing anything. Is there an alternative to the File.Copy method of copying files so I can report on the progress of individual file copies, like the percentage complete?
Looking for an idea of where the best place to keep files for a program. I use them frequently and they interact with the program directly. I want to place them in a place where they won't be easy to find if someone is just cruising thru the directories. Any ideas?
I'm new in vb.net (application). I need to access files or browse files using vb.net. Does anyone knows how can I do it?? I can use the listview??This is the process:I have a form and I need to browse folders and drivers (C drive/network drive...) in the same form. But how can I do that, how can I display the icons, file, etc. and open it in listview or any control in vb.net.
Is there a way to get a list of files in a certain directory and then put them in a context menu strip then when they are clicked have the program open the file. I'm trying to make a flash player that can show the list of flash files and when you click the file path on the context menu strip it will play from that file. Here is what I have:
Private Sub Form1_Load(ByVal sender As System.Object, ByVal e As System.EventArgs) Handles MyBase.Load OpenFileDialog1.Filter = "Shockwave Flash Objects (*.swf)|*.swf|All files (*.*)|*.*"
I should mention that I'm mainly concerned with Random files.Which is the best method?
1.) Open file, manipulate data and immediatly close the file. (Which means doing this alot everytime data needs to be read, edited, deleted, or saved)
(2.) Open the file when the app starts and let it open for data manipulation until it's time to close the app at which time the file would be closed.
I was always taught option 1. The main reason at the time was to cut down on the chance for data corruption in the file or an outright loss of the file due to outside circumstances like power failure.
I'm programming a small windows app that allows the user to fill out some fields and then creates and runs a batch file based on their entries. The program works, but it doesn't.The program creates the batch file just fine but when it tries to run the batch file, it gets errors in the cmd window saying it cannot find the scripts the batch file is trying to call. What's confusing me is that if I take the batch file the program created and run it myself, it works just fine so I know it is not the batch that is the problem. My best guess is that there is some security setting for vs.net that I've overlooked but I am only an "every once in a while" programmer so I really don't know where to look. Additionally after running the batch, the program is supposed to delete the batch file but that fails as well with the message that it cannot find the batch file, the same batch file that it had just created and run.
have just started vb .net and im working with the 2005 edition, but i would now like to make a program that i can use at home that will alter files names,Bassically where i rip cd's to put them onto my mp3 player they are always given a name starting with 01. etc, what i would like is a small bit of code that uses a while or for loop to go through every file in the folder and remove the first three characters.
how to read and write random access records in .NET. Now I know random access files are dead and the response on every query I found with google/bing/yahoo searches was don't use random access use Binary serialization but I can not. I have a situation where I need to read and modify 256 byte records in a file that is used by another application.
Now for a bit of background the file is made up of 27 fixed length records of 256 bytes. Each of the records has a separate structure consisting of 30-120 fields. The proplem I am having is I can create the record structures but when using the FileGet method I am receiving an error. Now I have found samples that showed the same format I am using that claim to work.
The error I am getting is Option Strict On disallows narrowing from type 'System.ValueType' to type 'ProgramName.ModuleName.Record' in copying the value of 'ByRef' paramater 'Value' back to the matching argument.
The basic layout (simplified) is as follows:
Code: Structure Record <VBFixedString(6)> Public SomeParam as String Public SomeNumber as Short
[Code]....
Now the overloads for FileGet show all of the data types including Object. But I have tried various methods to retreive the 256 byte record but without success. Do I have to actually read each of the several hundred fields of the structure one by one? Or am i overlooking something (I have been known for this in the past)
Or (and I'll kick myself if this is the only way) do I just have to set option strict Off in the module and the code as above will actually work.
I would love to be able to just ditch the random access file but I have to keep it for compatibility with several other applications at this time.
Alright this is what im working with. Im Working with vb.net 2003.I have (X) amount txt files in "C:Marble".In each txt files there are 3 lines.I want to read the first line of each txt file and display them in a listbox. [code] Things i need to Find:
-1 How to tell how many files are in a folder
-2 How to create a loop were it reads the first line of each file in that folder
An application that my company uses stores report files with a .dat extension for use with their software. These files can be opened in notepad and the basic report information can be seen as text. I am attempting to write an app that opens and reads data from these .dat files and displays it in a checked listbox. My hangup thus far is that I can't figure out how to read the data from the .dat files since they are not .txt files. Can anyone point me in the right direction?
I know how to get a list of files in a local Windows machine's specific directory:
Dim dirInfo As DirectoryInfo = New DirectoryInfo("C:Temp") Dim fileListing() As FileInfo = dirInfo.GetFiles()
But how do I use this class for a mainframe system where user ID, password and 'change directory' command are needed?
When I attempt to just access the server without credentials and just do the Windows version and use the syntax "\server_nameAUTO:RPTS.", where 'AUTO:RPTS.' is the mainframe directory name I get the following error: "The given path's format is not supported."
When I just use '\server_name' I get: "The UNC path should be of the form \servershare."
For a project, I need to extract all the frames of an AVI video. This I have accomplished by with free program. The problem: it took a 30 second clip and spit back 5400 some frames, and the image only changes every 20 frames or so. I really only need about 270 frames, but I didn't want to go through each one manually. So, I wrote a small program using Visual Studio 2008 and VB to open each image in binary and compare it to the one before that. If there is a discrepancy of more than 55000 (to accommodate slight errors in capturing), it leaves the file be. If not, it deletes it.
After much troubleshooting, I got it all to work. I ran it on the directory containing the images, but when it finished, only the first image remained. I checked into it, and found that the FSO loop I have set up to loop through each file was pulling the files in apparently random order. It will give me the image "bg(01).bmp" and then immediately "bg(1362).bmp" and so on. And the order changes every run.
I tried setting up an array to generate a list of all the files and then sort them, so I could pull off the name of the file, then the next one, ect. but it took over 3 hours to finish processing the array, and then It was using a huge chunk of RAM.Is there some way to "force" vb to pull off files in alphabetical order? here is my, code, it belongs with a form. fPath is a global variable, simply pointing to the source directory.
