Structure For Large Quantities Of Data?
Jul 3, 2010
I have a large amount of data (400 variables recorded 2 times per second) which is output from a factory monitoring system. It is stored in encrypted files in blocks of one hour from which i can extract to csv with a small piece of software.I would like to upload these csv files into a single database from where i can make graphs, analysis or whatever over longer periods.
The csv files have one line of data per 'time stamp', each of which can be conceptually segregated into approximately 15 individual pieces of equipment.I can quite easily import the data into a single table. However, i find that, very quickly, i end up with an enormous file (about 700mb/month) which is extremely ponderous to work with.
is it possible to split the table into 15 separate tables related by 'time stamp' and second, can each of these separate tables be saved in separate files in storage memory (such that only the files with the desired data are imported into ram).
View 13 Replies
ADVERTISEMENT
Jan 5, 2010
I'm new to VB 2008 after having spent a long time with VB6, so I apologize if this is a stupid question. But I'd really like to have this straightened out.
Let's say I have a pretty large structure that has lots of properties.
Code:
Now say that I want an internal database with about 10 instances of this structure total, describing, say, 10 different products that a store sells. When these values are loaded from a database, they remain totally static. (However, they can be different each time a program loads)
Now say that I have a class. Each instance of this class is a type of that BaseProduct structure. Meaning, each instance of the class pertains to one of the 10 types of products that the store sells. However, this class has additional properties that pertain specifically to each instance, which are not static.
Code:
Now, the problem here is... If I have 200 different transactions, each one contains an instance of BaseProduct. BaseProduct is HUGE, and is largely redundant (only 10 types possible), so I think it's a little silly to include a whole copy of it with EVERY transaction. However, the Transaction class really needs information regarding the base product it pertains to. Is there a way to, instead of declaring a New BaseProduct in the Transaction class, to simply make one of the properties of the Transaction class a pointer to a BaseProduct variable?
In VB6, I would accomplish this by making a BaseProduct(10) array, and then giving each Transaction an ID number referring to an entry in that array. But in VB 2008, using class structure, this is impossible. I can't define the BaseProduct(10) array outside of a class in a namespace, and if I define it in the actual application's form, then the class loses modularity since it relies on the application that's using it.
View 11 Replies
Mar 24, 2011
I am trying to make a calculator to calculate how many sheets of a product of varying thickness to put into a stack and how many stacks are required to make a whole pack, taking into account that some packs have no cover sheets and some have one at the bottom only and some have top and bottom.
As you can see there are a few variables and to make it harder is the way the stacks are made. No stack can be over 180mm and the last two stacks are halved to make the last two an even height (roughly).
I have a For Next loop and this would be okay if it wasn't for the 2 odd stacks at the end and the cover sheets.
This is what I have anyway.
CB = txtBottomCB.Text
lstDisplay.Items.Clear()
MaxBookHeight = (txtBookHeight.Text)
[Code].....
View 5 Replies
Mar 16, 2012
In VB.NET I would like to create a complicated data structure with multiple types of data stored in an array like format (see below). I am trying to create a data structure that would look something like this: [Name; xLoc; yLoc; zLoc; [Jagged Array]] Note: Name needs to be dimensioned as a string, xLoc and so forth as integers. The Jagged Array would look like this:
[Code]...
View 1 Replies
Jun 9, 2011
I want to write a program to do Markov chain, but my states are quite large. First of all I calculate all the transition probabilities and revenues for all states(1381860 total states), and store in a multidimensional array. Public RevArr(0 To 9, 0 To 750, 0 To 282) As Long
After that the iteration of markov chain should use these as inputs to calculate the steady-state probabilities. But when I try to run the main code I got this error.Exception of type 'System.OutOfMemoryException' was thrown.
The following is the declaration of second array I add just another dimension for storing all the iterations, but I get this error. Dim stateprob(IT + 1, 0 To 9, 0 To 750, 0 To 282) As single
View 1 Replies
Aug 6, 2010
I have a large data.DataTable and some formatting rules to apply.For example, the LASTNAME column has a value of "Jones" but my formatting rule requires it be 10 characters padded with spaces on the right and uppercase only. Like: "JONES "My initial thought is to loop through each row and generate a string. But, I wonder if I could accomplish this more efficiently with a DataView, LINQ or something else.
View 1 Replies
Aug 6, 2009
I'm searching the same title "Large amount of data in ComboBox" last 1 and half years but I could not get what I'm expecting! Here is my case, I have 20 thousand unique products, I just fill into dataset & bind only two columns in the combobox for searching & validating the product if it is exist in the product list or not.
1) I read about dataset but all articles are described "Dataset only for few records because it is sitting in-memory". If I fill few records (100 or 1000 products) into the dataset, how can I search & validate REMAINING (NOT FILLED) products?
2) Is there anyother way without filling into dataset and bind to combobox? Can anyone suggest me which is suitable for me?
View 14 Replies
Sep 13, 2010
i want to display a large data in a cell of DataGridView as multiplelines to avoid the horizontalscroll.
View 1 Replies
Sep 21, 2010
I am working on a program for a customer and am working in VB2010 in VB language. I have two rather large files that contain data. File #1 is over 300,000 lines of data. File #2 is over 100,000 lines. The files are CSV text files. On a User Form, there are two values, "StartTime" and "StopTime". This is the process I am doing:
1. Read both files into ArrayLists
2. Get item #1 of ArrayList #1 (we'll call this "A")
3. Get item #1 of ArrayList #2 (we'll call this "B")
[code]....
View 6 Replies
Mar 2, 2010
I have a listview control in my form which contains 44 columns. When user clicks on particular node in the 'tree view' it loads corresponding records to the listview. If the number of records is less then it works fine, but if there is more number of records then it takes more time to load(very slow).
View 4 Replies
Jul 22, 2009
I'm working on a project at the moment, where all the data being loaded is kept in an XML file which until now have been rather clean with structures such as
Code:
<Textures>
<Texture id="1">TerraingrassDirtNW.png</Texture>
[code].....
View 2 Replies
Jun 22, 2010
I have a list of a few thousand names. I actually have last name, first name, phone number, etc and more info. I'm going to make a class to contain the data. so I'll have like a people object with for example a last name field. I'm going to store these in like an array list. Lets say i have 100 items in array list. Maybe last name smith is on index 3 21 and 63. I'd like a data structure where i can easily search for smith and get indexes 3 21 and 63. Brute force is simply loop through complete array checking last name and grabbing index each time i find smith. Ideally id like to have more efficiency. In c++ I'd use a multi map but i don't see that that exists in vb.net. A dictionary wont work because its 1:1.Name of data structure, short description of how to use it and web help reference page would be nice.
View 2 Replies
Feb 14, 2008
I would like to use data structure in VBA. the code like
Public Structure employee
Public firstName As StringPublic middleName As String
End Structure
But there was compile error: Expected :end of statement. Can I use data structure statement in this way? how to use data structure in VBA?
View 4 Replies
Jan 31, 2011
I tried to follow this advice, but I still run out of memory.
[URL]
Basically, I am reading all the bytes of a 1 gb disk and I want to try and do a md5 hash on it. However, when following this example, my cryptostream gets too large a System out of memory exception.
Is there another way to do the md5 hash over some much data?
I am currently reading it into 10 million bytes arrays at a time. Can I hash the first array, and then use it as a seed for the next array, and so one, like you might do to a CRC32 calculation?
View 2 Replies
Jan 7, 2011
I am trying to show a large amount of data in gridview but the problem is that everytime data increase the gridview row size increase automatically.
View 4 Replies
Sep 5, 2009
Is it possible to create a kinda virtual storage in a vb application so that data created during run time are stored right in the applicaion instead of its environment?
View 11 Replies
Feb 28, 2012
Some of my users are collecting data files this size and wish to load it in to my tool. Is there a way to load large files of string data w/o getting a OutOfMemoryException? I have tried many different storage methods (Array, List, etc) but always without success.Before anyone asks 'why load it all at once', let me say the nature of my app is speed critical and it must be so.
View 3 Replies
Feb 7, 2011
This is a problem with MS Access, but I can't seem to find an MS Access Forum, so I'm asking it here.I have a large table of just one field. The field type is Byte. I use CTRL C and CTRL V to make a copy of the table.I then compare the Source and Destination tables and lo and behold they are different at about record number 8 million.I also tried an Append Query to append to an empty table (I haven't tried a Make Table query).But, similar corruption.The source and destination tables do not end up the same. The Record where the failure occurs varies, but it is always greater than 8 million.I tried this on MS Access 2002 and 2010 (both with an Access 2002 database).
View 1 Replies
Nov 25, 2009
I have some structures and I change the data inside a loop. This works ok. But as soon as the method ends, all values in the structure are reset???
Heres my
Private Sub ReadDevice(ByRef device As Device)
Console.WriteLine(" Reading device " & Device.name)
Dim data As Short
[Code].....
View 5 Replies
Jun 24, 2012
I ran into a problem where the listbox is taking awfully long time to show up. This is not the same in java(It took very very less time - one second max). So, is there any remedy to this??, any other form control??, or some other way?? imports System.Threading imports System.Windows.Forms
[Code]...
View 1 Replies
Dec 1, 2009
So, I have a little issue.. I can't currently explain it using the software I'm developing because it's "internal"... So I'm going to describe it as an email serverAnd the email server has a fair number of email accounts, and I want to save the accounts emailFor development, I've been saving it all in a list of a structure (this sounds worse than it is, the data actually is no more than 120 bytes per "email")
View 5 Replies
Mar 12, 2011
I am trying to develop an application that will let me do the following
1) Be able to search to large binary files for a specific byte array (header) and to continue searching for another byte array (footer). Then i want it to copy the section of the binary file between these 2 locations to a specified file. Then the program is to continue searching for any other header/footers An example would be search a large file and carving out the jpgs it finds etc. This routine would have to be able to handle raw images of 20gb or more if needed.
View 10 Replies
Apr 29, 2010
How would I use the BinaryWriter to append binary data to a file that is very large? Say a file that is 3GB in size. The BinaryWriter's write method will only accept an integer for the index value. I've attached a screen shot of what I'm talking about. If I were to enter a value that exceeds the size of an integer on my 32bit Windows XP system, which I believe is 2147483647 it will reject it right?
View 4 Replies
Apr 25, 2011
I recently wrote a program that collects information from the network can displays it in a DGV for the user to see. I tested this on networks where I would end up with several thousand items brought back and all worked fine, very little memory increase and once the search had finished the memory usage went back to pretty much what it was before the search, so I assume I have no major memory leaks that I should be concerned about (I'm careful to always dispose of things and clear large variables where possible).
However, a new company has now started using my application and they have several hundred thousand items that my program will find. They reported that they leave the program running its search for a few hours and most of the time it crashes with an Out Of Memory exception. Now I did have my app do some caching of certain information to avoid querying the network more than was necessary so I thought well maybe this is what is causing it but I added an option to turn that off and they say it still happens. So the only thing I can see it being is simply the amount of data that the program is finding and having to store in memory.once my application has gathered information about the items it stores them in a List(Of MyItemClass) and then when the search is complete the items are added to the DGV. So I'm thinking maybe its just the size of this List(Of MyItemClass) that is simply getting too large. Bear in mind that each instance of MyItemClass in this List can have quite a lot of information in it as it has several properties that are List(Of String) that may contain a few thousand strings in some scenarios.How do you deal with such large amounts of data without running out of memory?I'm assuming the only option I have got is to 'page' some of the data to a temporary file on disk once I get over a certain number of items in the List(Of MyItemClass)?
View 2 Replies
Nov 6, 2011
I'm designing a choice based adventure game in VB.NET 2010, where you are presented with a story label, and you choose between 2 buttons, which choice you want to make. I want to know the most efficient way to do this.My goals are to store the labels for the buttons and the story in some form of data structure, which is temporarily a hash table right now, and some form of structure for the choices. Right now I am using a custom class that references the indexes of the next 2 choices and their respective label indexes which stores the class instances in a hashtable.I've looked into things like arrays, dictionaries, lists, and collections, but I'm not sure which one best fits what I'm after. Any .NET data structure would work. What is the most efficient data structure for these 2 pieces of data? Would just a string array work?
View 2 Replies
Apr 16, 2012
I have problem about how I call a function on DLL to get data from a structure. Have a C++ example about how it's works the library, and I want to use it on vb.net. his is the working c++ example:
Declaration:
BOOL (WINAPIV* MyFun)(DWORD start1, DWORD stop1, P_RESULT pResult, PDWORD pTpNum, myCALLBACK lpFunc) = NULL;
Call:
// pStruc = RESULT structure
// TrpNum1, TrpNum is ULONG
ULONG TrpNum1=0;
MyFun(Start,Stop,pStruc+TrpNum,&TrpNum1,&myCB);
Callback:
void __stdcall myCB (RESULT *pStruc) {
printf ("%.3f", (double)pStruc->val1);
tpn++;
[Code] .....
The error is this: "Attempted to read or write protected memory. This often indicates that other memory is corrupt."
View 1 Replies
Feb 7, 2011
Is there an easy way to get data from a table in SQL Express and put it into a Structure? The table has one string column and five bit columns. I'm using VB.net
View 13 Replies
Feb 19, 2011
I'm creating a Data Layer for a Web Application and in one of my functions, I need to pass in a Structure full of Customer info. This Data Layer is essentially a DLL. My problem is that I'm not sure how to do this.Below is my code and what I want to do.
[Code]...
View 5 Replies
Mar 7, 2010
Is the integer data type is structure? And if yes what is the original code of it?
View 2 Replies
Jun 6, 2011
im tryin to store data in an array using a structure. However, the code im using keeps returning a null referece exception, even when there is something in the input boxes. This is the code im using:
With SaveInfo(Index)
.EmployeeID = EmployeeIDComboBox.Text
.EmployeeName = EmployeeNameBox.Text
[Code].....
View 6 Replies
