Load Large CSV File Into Database?
Oct 14, 2009
I'm trying to load a large CSV file into a Microsoft SQL Server Compact 3.5 database. I've tried using the following: Using MyReader As New Microsoft.VisualBasic.FileIO.TextFieldParser("filename.txt") MyReader.TextFieldType = FileIO.FieldType.Delimited MyReader.SetDelimiters(",")Then splitting the data with MyReader.ReadFields() etc, before using this data to add rows to a dataset in a database table in my project. However, my CSV files are very large, at above 9.5 million rows, and this takes forever, if the computer doesn't crash also. Does anyone have a better idea for what to do? I would like the CSV file to be loaded into the database table, to enable me to sort it, and undertake some querys and maths. The CSV data structure is:2,193,761.40000000000012,43,1510.22,8,1929.60000000000012,22,2564.52,22,2791.70000000000032,19,2971.6000000000004
View 3 Replies
ADVERTISEMENT
Feb 8, 2011
I have 4k records in access database. And one of the field value contains ~100 lines each so and one other field has ~25 lines. So total database size reaches ~30MB and it takes lot of time 15-20 seconds to load the database in vb.net using odbc [URL]..and updating of any other small fields also takes time due to database being large
So as an alternative I used rtf file (txt files were not preserving all the newline characters). So these file are around 5-10kb only. But for 4k records and 2 fields I have now 8k files. And copying of these 8k rtf files is taking huge time for 5MB transfer it takes an hour or so.So is there any other alternative for storage of this data. So that it will be portable and easily loaded/accessed/updated from vb.net?
View 3 Replies
Oct 28, 2010
I need to load a large txt file that is in a fixed width format. There are over 45K lines, so speed is important.I need to load one of the fields into a dropdown box and have another field (label) display the text of another field in the related line.I could import the file to an access db if needed, but would rather not as i also want the txt file to update from a link on a regular bases. So having it in a DB would be more work to process that part.[code]
View 1 Replies
Sep 22, 2011
I have to export the data from the database to .txt file, I have the code below that does export it to .txt file but I'm having problems with this code because the data that I'm exporting is about 5 000000 records, it is quite slow and sometimes the pc can't handle it and stops the exporting.
Dim saveFile As New SaveFileDialog
If saveFile.ShowDialog = Windows.Forms.DialogResult.OK Then
command = New SqlCommand
[Code]....
View 5 Replies
Oct 15, 2009
ok so i currently have a large image loading in to a picture box the image is 13424 x 11344 and 8.10mbs and it up around 460,000k of memory i was just wondering if they was a more efficient way of loading this image?
View 4 Replies
Oct 30, 2010
I want to load minimum 150MB of XML into dataset,
simple way of doing that is
XmlDataDocument xmlDatadoc = new XmlDataDocument();
xmlDatadoc.DataSet.ReadXml("C:\books.xml");
But i feel on Lagre size message this code make some performance issue.
View 3 Replies
Aug 8, 2009
I want to load minimum 150MB of XML into dataset,simple way of doing that is
XmlDataDocument xmlDatadoc = new XmlDataDocument();
xmlDatadoc.DataSet.ReadXml("C:\books.xml");
But i feel on Lagre size message this code make some performance issue.
View 12 Replies
Sep 5, 2011
I want to load 2 crore records in a treeview control with parent child Hierarchy order
for example
- Groupa
- a1
- a2
[Code[....
View 1 Replies
Nov 9, 2010
I wrote last week about a problem with an MDI program that had a large number of forms (each with a large number of controls on it) that was "sluggish" in loading and in switching between the child forms.I've attached an example program in VS2008 (though the actual app is VB2010)Rather than show the hardware control application with all of the text fields, and picture boxes acting as indicators, I made a simple program to show the point. This is exaggerated as this just loads 2000 or so text boxes on a form. In the real app I have ~ 200 assorted controls per form of picture boxes, scroll bars, text boxes, labels etc. Also, there is a large full screen .jpeg as the back ground of each form. All are generated at run time and are placed on the form in the New call of each form. (as in the sample attached)main issue seems to be the methodology I use to switch between child forms. I make the current form .visible = false, and the next.visible = true. I have used this as it "keeps the place" on each page if the user has scrolled or is looking at one section of the form. When the next form .visible = true happens I see the controls added in a "machine gun" fashion instead of all at once.
View 28 Replies
Jun 15, 2010
how to load a file in combobox?how to add, remove and search contents in this file? n how to use filters?
View 1 Replies
Sep 13, 2009
Is it possible to use a file browser to load a connection string when the start form is loaded?
Using ofb As New OpenFileDialog
If ofb.ShowDialog = Windows.Forms.DialogResult.OK Then
Dim cConnectionString As String = ofb.FileName
[Code]....
View 7 Replies
Mar 12, 2009
how to load a .bak file of database backup in sql server 2005 using vb.net 2008 windows application?
View 5 Replies
Mar 13, 2012
I'm faced with a (fun, actually) data mining problem; I have raw ASCII files from instruments, and I want to move that data (~100 GB) into a database, then be able to run various algorithms determining correlations of time series, etc. I would like to write the mining algorithms in Visual Basic (.net, VS 2010 right now), and be able to do visualizations with VB code I have in hand.On the nature of the data: think of a set of several thousand devices, each recording a measurement at a given interval - so I'm talking time-series vectors.
I spent today re-acquainting myself with VB.NET's interface to (in one case) a Microsoft Access database. What used to be fairly simple - DAO I think it was - involved tables, recordsets, etc (and that would likely be fine). Now I seem to be required to have a weird variety of generally useless objects, e.g. 'adapters', 'datasets', etc. The problem is that I know exactly what I need, and all this extraneous stuff just gets in the way (certainly in coding complexity and opaqueness, and likely in efficiency as well).If any of these mechanisms gave me a kind of virtual access to the entire dataset, and let me control caching parameters, etc.it might be great, but I found nothing along those lines.
Anyway, I tried a number of different approaches, and none seemed at all aimed at what I need to do: efficiently do math on a large dataset. I can't believe I'm the first to have this problem, but I can find no useful wisdom out there. I'd be comfortable with pretty much any underlying database mechanism: MySQL, SQL Server, MS Access, but ideally something generally SQL based (I may eventually have to transition this entire system to draw from a client's SQL database, though that's not an overriding concern now). Other than that I want simplicity and efficiency. I thought my old ODBC techniques would work, and to some extent they do, though modifying tables seemed to have bizarre problems (no errors, but not modifications either).
I do have a fairly aggressive deadline to show some algorithm results, so my focus in the short term is to get something reasonable working *in* the short term - in other words, it's less important to me to pick the 'fastest' relational database than it is to pick a database that lets me focus on coding the algorithms, not working through tedious data access coding. If this db could be any smaller, I'd have tried to do it all 'in memory' at least for proving concepts; I don't want to have to learn an entire jargon and approach just to be able to retrieve data points, and write out new info (e.g., first derivatives of time series).
Perhaps I'll need to bite the bullet and just write something myself, a .dll perhaps just to save and restore large time series vectors. It seems a bit frightening to me that one would have to do this in this age, what with all the database systems out there, but I don't have much time to work through arcane interface logic.
View 6 Replies
Dec 27, 2009
I am currently helping to write a Visual Basics program that was originally writing with access.The Original Access program has been used and modified over the past 5 years, During that time the database has gotten pretty big.The problem I am having is If I add the entire database to my data sources it creates data tables for every Table and Stored procedure even when they are not currently used in the program resulting in a 20+ minute build time just to debug.I am currently only adding Tables and Stored Procedures as i need them, so its not that much of a problem now, but as I continue to work and add tables the build time is going to go back up, and Id hate to have to wait 20+ minutes to test a small tweak I made.
View 1 Replies
Feb 8, 2010
[code].....
View 1 Replies
Feb 21, 2012
The Adventureworks database is used extensively for tutorials in Visual Basic applications. This has many tables that can be bound to a solution. What I would like to know is, when using such a large database with many tables and using lots of them, is it better to construct your program with a number of projects all under one solution or to put everything into a single project. If you use a number of projects, what is the syntax for accessing each module from a menu strip in your main form? I know how to do this in a single project - all you do is Form(x).show. In Adventureworks, you have a table for Employees that could be put into a single project with a number of forms - one for data entry, one for displaying all of the employees, a filter to a single employee, etc., etc.
If two small projects rely on each other for some reason, again the syntax for linking them could be an issue. Would it be better to combine them into a single project? My reason for asking is that I would feel safer if I created an application on a project by project basis and then brought them all together in one solution. This way they can be tested individually and then linked together where necessary. Making a disastrous mistake in one then only requires a small re-write rather than the headache caused by a total crash of the whole solution. I would like to know which option professional developers take. I am using Visual Studio 2008 Professional and SQL 2005 Express
View 5 Replies
Apr 22, 2011
I'm developing a VB.NET application to obtain data from any data source (using an odbc connection string), and because this i can't use specific .net connectors like MySql.Net connector and i can't previously know if the file/DMBS supports LIMIT parameter. This app will read data from a table in the data source and use the information to make some files. At this point there was no problem with some tables because they are less than 3,000,000 records length, but there is a table that has 5,000,000+ rows length, and when i send the query the connection is lost. I'm working with OdbcDataReader, because i read on MSDN site that this is the best way to read a table one row at time and i just use each record once.[code]...
View 2 Replies
Jun 9, 2011
I want to write a program to do Markov chain, but my states are quite large. First of all I calculate all the transition probabilities and revenues for all states(1381860 total states), and store in a multidimensional array. Public RevArr(0 To 9, 0 To 750, 0 To 282) As Long
After that the iteration of markov chain should use these as inputs to calculate the steady-state probabilities. But when I try to run the main code I got this error.Exception of type 'System.OutOfMemoryException' was thrown.
The following is the declaration of second array I add just another dimension for storing all the iterations, but I get this error. Dim stateprob(IT + 1, 0 To 9, 0 To 750, 0 To 282) As single
View 1 Replies
Aug 26, 2011
this code takes about 30 mins and high cpu usage, what is the problem
Do
strLine = objReader.ReadLine()
If strLine Is Nothing Then
[code].....
View 3 Replies
Feb 8, 2009
use the following code to create a database in ms access
Code:
'reference: Microsoft ActiveX Data Objects 2.8 Library
'reference: Microsoft ADO Ext. 2.8 for DDL and Security
[code].....
View 1 Replies
Aug 26, 2011
this code takes about 60 mins and high cpu usage for a text file of 90,000 lines, what is the problem..[code]
View 9 Replies
Apr 4, 2012
The file is a UTF8 text files.
Each characters has varying number of bytes and each line has varying number of characters.
Does vb.net has table of line numbers to byte location function or something like that?
Also after that how to read that?
View 1 Replies
May 29, 2011
I am writing an application in VS2010 using VB.net.The application is relatively simple and I would expect the .exe to be less than 1 Meg.I have written a couple of applications that are quite a bit larger that are less than 2 Meg.I am compiling in "Release" mode.The file size is 29,127 KB (in debug mode it is just 29,167 KB)Where do I start looking to find why the .exe is so large?
View 2 Replies
Feb 22, 2012
Base on the related to the last thread please you clarify of how to load song fles to database that we have been set ?
View 12 Replies
Dec 17, 2008
I have problem, how I can import HEX file to my program in VB .NET? And I have question if I can import, What size of file I can import? I have 200MB - 400MB HEX files?? Can I import so huge files?
View 18 Replies
Mar 16, 2010
I am trying to read this XML document.
An excerpt:
<datafile xmlns:xs="[URL]" xmlns:xsi="[URL]" xsi:noNamespaceSchemaLocation="wiitdb.xsd">
<WiiTDB version="20100217113738" games="2368"/>
<game name=" Wanted: 50 Wacky Jobs (DEMO) (USA) (EN)">
<id>DHKE18</id><type/>
<region>NTSC-U</region>
[Code] .....
It just skips the "While iter.MoveNext" part of the code. I tries it with a simple XML file, and it works fine.
View 1 Replies
Jan 5, 2012
I have a simple app that reads from a very large text file, and returns a value if a string is found. I can instruct users where to download the file, and where to put it, but it would be nice if I could embed the file with the publish, so that the program knew where to look by default.Getting users to download a seprate file is painful. This file has 1.4 milion lines of text. I really need it to look for the file in a predictable place and be able to run against that by default for most users. I can have experienced users browse for a new file, but most people aren't into that much thought.
View 8 Replies
Oct 7, 2011
I need to read the second-to-last line of a very large log file.
I can't read the entire thing into memory, count lines, etc etc. I can't use Filestream.Setlength because that needs readwrite access and the log will be opened by another application. And it has to be fast. However, the line ends in a cr/lf meaning the last line is actually empty. Been struggling with this all day, and its hurting my head! Not good on a Friday!
I have a function that can read the last line but can't get it to go one line up. I can get it to read characters from the end of the line, but that's not much of a help with a variable length line!
Maybe fs SeekOrigin would work as it could run backwards looking at an example from MSDN - need to get the data before the last cr/lf and end at the next one... hmmm... problem is that that example also writes the text backwards as well.
I'm monitoring the log for particular entries for issues that are causing us grief at the moment.
View 5 Replies
Jan 30, 2012
My problem is I have very large text files (approx 2GBs+).They have records in them based in one per line.Each line is not the same length and the data can be different lengths all the time.I am currently reading the file line by line, then splitting the data by common characters in the records. To process the full file it currently takes 3hours. This is way too slow for its purpose.
View 4 Replies
Jul 6, 2011
I've a problem reading text file using StreamReader. The file have between 500 000 and 1 000 000 lines.When I try to read it in a cycle, I get an error. That's why I've tried the StreamReader.ReadToEnd method. It worked fine. I've get the entire contents of the file in one string. So far everything is okay, but I've a small problem searching this huge string. I have to reformat the string to my desired format. I'll try to be more specific: The format of the input file is as follows:
50471100 8 2 6 5 0<LF><CR>
00000016 365442 12231<LF><CR>
00000026 112166 31133<LF><CR>
<end>
[Code]...
View 7 Replies
