I'm faced with a (fun, actually) data mining problem; I have raw ASCII files from instruments, and I want to move that data (~100 GB) into a database, then be able to run various algorithms determining correlations of time series, etc. I would like to write the mining algorithms in Visual Basic (.net, VS 2010 right now), and be able to do visualizations with VB code I have in hand.On the nature of the data: think of a set of several thousand devices, each recording a measurement at a given interval - so I'm talking time-series vectors.
I spent today re-acquainting myself with VB.NET's interface to (in one case) a Microsoft Access database. What used to be fairly simple - DAO I think it was - involved tables, recordsets, etc (and that would likely be fine). Now I seem to be required to have a weird variety of generally useless objects, e.g. 'adapters', 'datasets', etc. The problem is that I know exactly what I need, and all this extraneous stuff just gets in the way (certainly in coding complexity and opaqueness, and likely in efficiency as well).If any of these mechanisms gave me a kind of virtual access to the entire dataset, and let me control caching parameters, etc.it might be great, but I found nothing along those lines.
Anyway, I tried a number of different approaches, and none seemed at all aimed at what I need to do: efficiently do math on a large dataset. I can't believe I'm the first to have this problem, but I can find no useful wisdom out there. I'd be comfortable with pretty much any underlying database mechanism: MySQL, SQL Server, MS Access, but ideally something generally SQL based (I may eventually have to transition this entire system to draw from a client's SQL database, though that's not an overriding concern now). Other than that I want simplicity and efficiency. I thought my old ODBC techniques would work, and to some extent they do, though modifying tables seemed to have bizarre problems (no errors, but not modifications either).
I do have a fairly aggressive deadline to show some algorithm results, so my focus in the short term is to get something reasonable working *in* the short term - in other words, it's less important to me to pick the 'fastest' relational database than it is to pick a database that lets me focus on coding the algorithms, not working through tedious data access coding. If this db could be any smaller, I'd have tried to do it all 'in memory' at least for proving concepts; I don't want to have to learn an entire jargon and approach just to be able to retrieve data points, and write out new info (e.g., first derivatives of time series).
Perhaps I'll need to bite the bullet and just write something myself, a .dll perhaps just to save and restore large time series vectors. It seems a bit frightening to me that one would have to do this in this age, what with all the database systems out there, but I don't have much time to work through arcane interface logic.
I know this is probably a stupid question. I am converting a huge string to a integer, it is not working.....saying the number is too large to handle. It is only a 10k character string.I figured this would be fine because 2^16 power is 18446744073709551616.
I have some functions written in C that read & write large binary files (> 4 GB) using routines CreateFile, ReadFile, WriteFile & SetFilePointer, that use the HANDLE data type to access the file. Some others who are more familiar with Visual Basic would would like to be able to call my C routines to work with such large binary files. We particularly need SetFilePointer to jump around with the files.
I am currently helping to write a Visual Basics program that was originally writing with access.The Original Access program has been used and modified over the past 5 years, During that time the database has gotten pretty big.The problem I am having is If I add the entire database to my data sources it creates data tables for every Table and Stored procedure even when they are not currently used in the program resulting in a 20+ minute build time just to debug.I am currently only adding Tables and Stored Procedures as i need them, so its not that much of a problem now, but as I continue to work and add tables the build time is going to go back up, and Id hate to have to wait 20+ minutes to test a small tweak I made.
I'm trying to load a large CSV file into a Microsoft SQL Server Compact 3.5 database. I've tried using the following: Using MyReader As New Microsoft.VisualBasic.FileIO.TextFieldParser("filename.txt") MyReader.TextFieldType = FileIO.FieldType.Delimited MyReader.SetDelimiters(",")Then splitting the data with MyReader.ReadFields() etc, before using this data to add rows to a dataset in a database table in my project. However, my CSV files are very large, at above 9.5 million rows, and this takes forever, if the computer doesn't crash also. Does anyone have a better idea for what to do? I would like the CSV file to be loaded into the database table, to enable me to sort it, and undertake some querys and maths. The CSV data structure is:2,193,761.40000000000012,43,1510.22,8,1929.60000000000012,22,2564.52,22,2791.70000000000032,19,2971.6000000000004
The Adventureworks database is used extensively for tutorials in Visual Basic applications. This has many tables that can be bound to a solution. What I would like to know is, when using such a large database with many tables and using lots of them, is it better to construct your program with a number of projects all under one solution or to put everything into a single project. If you use a number of projects, what is the syntax for accessing each module from a menu strip in your main form? I know how to do this in a single project - all you do is Form(x).show. In Adventureworks, you have a table for Employees that could be put into a single project with a number of forms - one for data entry, one for displaying all of the employees, a filter to a single employee, etc., etc.
If two small projects rely on each other for some reason, again the syntax for linking them could be an issue. Would it be better to combine them into a single project? My reason for asking is that I would feel safer if I created an application on a project by project basis and then brought them all together in one solution. This way they can be tested individually and then linked together where necessary. Making a disastrous mistake in one then only requires a small re-write rather than the headache caused by a total crash of the whole solution. I would like to know which option professional developers take. I am using Visual Studio 2008 Professional and SQL 2005 Express
I'm developing a VB.NET application to obtain data from any data source (using an odbc connection string), and because this i can't use specific .net connectors like MySql.Net connector and i can't previously know if the file/DMBS supports LIMIT parameter. This app will read data from a table in the data source and use the information to make some files. At this point there was no problem with some tables because they are less than 3,000,000 records length, but there is a table that has 5,000,000+ rows length, and when i send the query the connection is lost. I'm working with OdbcDataReader, because i read on MSDN site that this is the best way to read a table one row at time and i just use each record once.[code]...
I want to write a program to do Markov chain, but my states are quite large. First of all I calculate all the transition probabilities and revenues for all states(1381860 total states), and store in a multidimensional array. Public RevArr(0 To 9, 0 To 750, 0 To 282) As Long
After that the iteration of markov chain should use these as inputs to calculate the steady-state probabilities. But when I try to run the main code I got this error.Exception of type 'System.OutOfMemoryException' was thrown.
The following is the declaration of second array I add just another dimension for storing all the iterations, but I get this error. Dim stateprob(IT + 1, 0 To 9, 0 To 750, 0 To 282) As single
I have several installations of my Linq-to-Sql app running in the field. Now I've created a new version, which adds a new column to a certain table. I've added this column in the dbml file. But when updating the installation, I want to preserve the existing database. How to handle this? Linq-to-SQL doesn't seem to like this inconsistency.Is there an easy way to update the existing database using my new dbml file?
I have to export the data from the database to .txt file, I have the code below that does export it to .txt file but I'm having problems with this code because the data that I'm exporting is about 5 000000 records, it is quite slow and sometimes the pc can't handle it and stops the exporting.
Dim saveFile As New SaveFileDialog If saveFile.ShowDialog = Windows.Forms.DialogResult.OK Then command = New SqlCommand
I have 4k records in access database. And one of the field value contains ~100 lines each so and one other field has ~25 lines. So total database size reaches ~30MB and it takes lot of time 15-20 seconds to load the database in vb.net using odbc [URL]..and updating of any other small fields also takes time due to database being large
So as an alternative I used rtf file (txt files were not preserving all the newline characters). So these file are around 5-10kb only. But for 4k records and 2 fields I have now 8k files. And copying of these 8k rtf files is taking huge time for 5MB transfer it takes an hour or so.So is there any other alternative for storage of this data. So that it will be portable and easily loaded/accessed/updated from vb.net?
what's the best way to handle the updates to the database.For example i have one datagridview that's binded to a datatable (with two fields, id auto incremented and a varchar). The user can do all the operations Insert, Update, Delete.If i insert one row (only the text) and call the update method to save the changes to the database, and after that remove the same row and call the update method the adapter will throw a exception because it didn't update the datatable with the value of the id. So the best way to do this is always calling the fill method after every update?
I am planning to write an application which will be available in 2 versions. one free and the other one paid. The only difference is that in free versions only a maximum of 2 DB entries are allowed. but its unlimited in paid version
I am planning to write an application which will be available in 2 versions. one free and the other one paid.The only difference is that in free versions only a maximum of 2 DB entries are allowed. but its unlimited in paid version.DB does not have more than 1 table with 5/6 fields.I am planning to use xml for free version and SQLite for paid version. Is this ok? are there any other options?
using the sql server management studio we can create users to access the database. so we can grant several privileges as well as revoke some. so i want to create a login form with those user's. how to do it. is there any other way [code] in the connection i have to mention the user also.. is it possible. if its possible.
Whats the best way to handle the connection to a db when it's accessed often? I have an app where the user selects an item from a drop down list(call to the db) and then a picture of the item is returned along with it's relevant information(another call to the db). Small application but basically every action results in a call to the database. Should i just be filling like a global dataset and then accessing the information from there(if thats possible)? The database is stored locally(currently an Access Db but will be SQL in the future)
I have run into this problem a few times and have never come up with a good answer. I figure others must have dealt with this already.I have a datareader returned from the database and I want to use the values therein however values may or may not contain NULL. I would like to have a unction that takes in the value from the datareader and returns the value if it is not NULL and blank space if it is NULL.The problem I have is that the data-type of the variable I am testing is variable. It can be a String, and Integer or a DateTime. Can anyone suggest a simple way to test the value and then return the original value (as same data-type if possible) or something else if it is NULL
I am reading dates from a table in a database to display in a DataGridView column that is formatted as a date. The problem is that when the value is null in the database my program errors out because it cant convert the value to a date. How do you guys handle this? *Edit: If I set the value of my date variable to "Nothing" it displays as "1/1/0001" in my DataGridView, and I would like the cell to be empty for the null dates.
I just learned some basics of multi threading in VB.net recently as I came across processing a large List of records and inserting them into SQL database one by one.
I have code look like this:
Private Sub btnLoadNow_Click(ByVal sender As System.Object, ByVal e As System.EventArgs) Handles btnLoadNow.Click
Dim autoLoad1 As New Thread(AddressOf AutoLoad) autoLoad1.Start()
[CODE]...
Itemlists is a global lists i retrieved from database, everything looks fine to me but when i run this program, I found that the threads are inserting one records 9 times into the database(I have 9 threads in total), which makes me think that maybe i need to specifically assign 1/9 of a list to each thread? Is there another way to do it which doesn't require the splitting of the list, if not , how do i split it then assign?
I wrote an application to handle a joystick with directinput. It works fine if it uses the handle of the main window but it doesnt happen the same if the handle is the one of the component that contains the joystick's operation.
I mean:
main form handle: 15
component handle (which is inside the window and the joystick needs the focus of this component to work): 25
It always work if the handle i initilizate the device is the main form (15).How can i use only the handle of the component?
In CRUD application development we have to face the issues with concurencies in Ids such as Generate and display UNique EmployeeIDs on forms. Currently i'm writting .DLL file that has a class that will handle concurrencies.
But i'm puzziled with the Question, is this concurrency handle is part of the software developer or Database developer?
Exact duplicate: How to switch a large app from VB6 to VB.Net Exact duplicate: Best strategy for moving applications from VB6 to VB.Net Exact duplicate: Conversion tool comparisons for VB6 tool which converts vb6 to vb.net
So I am making a file crypter so that I can encrypt my VB.NET Application that I am making so that people can't decompile it. I had made this in C# and am transfering it to VB.NET, Everything worked fine in C# but once I had re-written the code in VB.NET i get this error inside of my RC4 Encryption method:
'Arithmetic operation resulted in an overflow.'
The error is occuring here:
Dim t As Int64 = (s(i) + s(j)) Mod 256
This is the same code above in c#:
int t = (s[i] + s[j]) % 256;
Is there anyway to make that calculation with it erroring? And why does it work in C# but not VB.NET?
provide Vb6 sample code to conert this value 2450258259404013568 to Hex as &h220110E7A7BE4000. while trying from win calc it's returning correct value as required.
How do you recommend I copy large files over the LAN (>20GB)Currently I am using File.Copy.Another question I have is how can I implement this class to be self contained, meaning that it would report progress to the object that called it? or do I have to define a delegate in the calling object?
Code: Public Class cCopier Private Delegate Function CopyProgressRoutine(ByVal totalFileSize As Int64, ByVal totalBytesTransferred As Int64, ByVal streamSize As Int64, ByVal streamBytesTransferred As Int64, ByVal dwStreamNumber As Int32, ByVal dwCallbackReason As Int32, ByVal hSourceFile As Int32, ByVal hDestinationFile As Int32, ByVal lpData As Int32) As Int32
