Find Distinct Lines In Large Datatables?
Oct 6, 2010
Currently we have a large DataTable (~152k rows) and are doing a for each over this to find a sub set of distinct entries (~124K rows). This is currently taking about 14 minutes to run which is just far too long.
As we are stuck in .NET 2.0 as our reporting won't work with VS 2008+ I can't use linq, though I don't know if this will be any faster in fairness.
Is there a better way to find the distinct lines (invoice numbers in this case) other than this for each loop?[code]....
View 1 Replies
ADVERTISEMENT
Jun 22, 2011
I have a DataTable "dt1" that has thousands of rows.
Columns are:
- Type_And_Model
- Val_1
- Val_2
- Val_3
- Val_4
There are only about 30 different "Type_And_Model"-types.How can I insert into a new DataTable (dt2) only distinct "Type_And_ Model" types with the sum of their specific values and total appearance number?
dt1 - DataTable with thousands of rows
Type1 2 2 5 5
Type1 4 2 7 5
Type2 2 2 5 8
Type1 2 2 9 5
Type3 3 2 5 5
Type1 6 2 5 7
Type3 2 7 5 9
Type1 8 2 9 5
.....
INTO =>
dt2 -DataTable with disinct types and their sum values + appearance numbers.
Type1 22 10 35 27 5
Type2 2 2 5 8 1
Type3 5 9 10 14 2
View 7 Replies
Feb 28, 2012
Some of my users are collecting data files this size and wish to load it in to my tool. Is there a way to load large files of string data w/o getting a OutOfMemoryException? I have tried many different storage methods (Array, List, etc) but always without success.Before anyone asks 'why load it all at once', let me say the nature of my app is speed critical and it must be so.
View 3 Replies
Jun 1, 2012
I'm having a hard time joining 2 datatables and have the joined datatable as result. First datatable (labels) holds data including a printerid. Second datatable (printers) holds printer references (id > unc). I would like to have as endresult (joined) a datatable with all data from the first datatable with the field (unc) of the second datatable. [Code]
View 3 Replies
Jul 14, 2011
When you need to do random access to the lines of a text file, the regular solution is to put those lines in a list and access them in that list.But doing this if the text file is large creates a large memory overhead and may cause some memory stress on the application.This class maps the file and provide the ability to read any particular line in the file at a random position, without the need to read or put in memory any other line but the desired line. The only memory overhead involve is the 1024 bytes buffer of the base stream.Even if the file mapping can be a process that consume a noticable amount of time if the file is very large ( ~ 5 seconds for each 100,000,000 caracteres) , once the mapping is done the access to a particular line becomes instantaneous regardless of the size of the file. (The file mapping can always be done at a convenient time (ex when the process load) and can be done on a worker thread The class is base on a StreamReader to support the differents Text Encodings Example uf usage ( As I say, if the file is very large, dont do it like that, but declare the class at a convenient time)
Imports Reader = System.IO.MyStreams
Public Class Form1
Private Sub Button1_Click(ByVal sender As System.Object, ByVal e As System.EventArgs) Handles Button1.Click
[code]....
View 11 Replies
Jul 13, 2010
I am trying to distinct on multiple columns and get datarows from datatable. but getting error.
Dim query As IEnumerable(Of DataRow) =
(From row As DataRow In SourceTable.AsEnumerable() _
Select row.Field(Of String)("ColumnName1"),
[Code]....
I want another datatable with distinct row based on given columns from SourceTable.
View 3 Replies
Jun 4, 2012
I am creating a VB application in Visual Studio 2010 that installs some SQL stored procedures. These stored procedures are encrypted otherwise I would just supply my clients with .sql files to create the stored procedures.
The install script that creates the stored procedure is 1500 lines in length and has been formatted in the way VS 2010 requires multi-line literals to be in:
"First Line" & _
"Second Line" & _
"etc..."
[Code]....
View 1 Replies
Aug 2, 2010
I have a huge text in a RichTextBox. There are few things in that text that I want to pick up and add in a ListBox. Each words I need from that text box are not identical but has got some identical keyword before and after them. Let me give you a sample...
"href ='displayCompany.php?name=BERGERPBL' class='abhead' target='_top'>BERGERPBL 958.00 <img src='tkdown.gif' border='0'><br>-25.30 -2.57%</font></span></a><br></td></tr></table></td><td
valign='top' height='30'><table width="120" border="0" cellspacing="0" [code]....
Here, I need to pick up the words, "BERGERPBL","BEXIMCO", "BEXTEX". They have got identical word "displayCompany.php?name=" before them and "' class=" after them.So how would I do it. I need a code in VB.NET. I use visual studio 2010. Remember, this is just a short sample. The real document is far larger.
P.S. Upper cases are not applicable. Because they may also be in lower cases.
View 13 Replies
Nov 1, 2009
I would like to step through each *.txt file in a given directory, open it, find and replace all, perhaps a thousand, occurances of a given string (e.g. "00/00/00"), then close and save file, and step to next one in the directory to do the same find and replace. I've been looking at filestream classes and regular expressions but don't see a clear way to do this.
View 4 Replies
Feb 17, 2011
Say I want to remove all lines that contain "hello". One way to do it is to read the whole files into memory and then operate. Often my files are big, 500MB. If I load them I got no memory. Will upgrading to a 64 bit computer helps? I have a big page file. Also I wonder how in the earth windows 7 and xp 32 bits can have very large page files anyway.
View 9 Replies
May 16, 2012
i have a txt file:
118.97.75.226:8080,
218.14.227.197:3128
118.96.151.110:8080
219.83.100.205:8080
this in an array, how do i find how many lines are in this array (obviously there is 4 but i want it to automatically detect it because it may not always be four if i import a differnt txt file)
Dim lines() As String
Private Sub Form1_Load(ByVal sender As System.Object, ByVal e As System.EventArgs) Handles MyBase.Load
lines = IO.File.ReadAllLines("fileName.txt")
Label1.Text = lines(0)
[code]....
View 1 Replies
Mar 10, 2009
Hi.
Im using this little function to search for text within some downloaded HTML source:
Public Function GetStringBetween(ByVal InputText As String, _
ByVal starttext As String, _
ByVal endtext As String)
Dim lnTextStart As Long
Dim lnTextEnd As Long
lnTextStart = InStr(StartPosition, InputText, starttext, vbTextCompare) + Len(starttext)
lnTextEnd = InStr(lnTextStart, InputText, endtext, vbTextCompare)
If lnTextStart >= (StartPosition + Len(starttext)) And lnTextEnd > lnTextStart Then
GetStringBetween = Mid$(InputText, lnTextStart, lnTextEnd - lnTextStart)
Else
GetStringBetween = "ERROR"
End If
End Function
It works great, but i need to be able to search for something that spans over 2 lines. E.g.
Sam works in a shop from the hours
of 09:00 to 15:00
I want to search for the words "hours of" - but because they are on seperate lines, it cannot find it.
I have tried "hours" & Environment.NewLine & "of" - and other variants of envirmonment.newline (knowing they all return the same value, but, hey, you gotta try) but to no avail. Surely there must be some sort of syntax to ge around this?
Any ideas?
View 5 Replies
Jun 8, 2011
How to find Lines of Code in VS 2010?
View 7 Replies
Aug 22, 2010
I need to go line by line through a text file as well as a listbox and remove lines over x amount of characters as well as adding the number of skipped lines to a label. I am trying to use a listbox in this example but it is not working correctly. What I am I doing wrong? First it will only remove an item when it is highlighted (focused) how can I make it so that is not required?[code]...
View 7 Replies
Jun 9, 2012
I know this is a very noob question but would it be possible to find a word in my coding and replace it with three lines?
[Code]...
So far I can only replace it with "ex1.Message) Finally Application.exit" but it's going to take me forever to go through my coding to change it. I'm not sure what character I could use to start a new line in between "ex1.Message)" and "Finally" and "Application.exit"
View 1 Replies
Aug 4, 2011
I have a program that will read a text file and put information from the text file into multiple text boxes. What I am trying to do now is be able to change the values in the textboxes and be able to write the specifc changes back to the text file. One way I though of doing that was using a while loop and copy the lines of the text file to another text file, then when it finds the line that has the specific change from the text box, make the change and continue writing the line until it reaches the end of the file. Then I was going to copy the file and overwrite the original, escientally making the change to the new file. Right now it will make the new file but will just make a blank text file. I think the problem is in the while loop statement but I am not sure. Here is the code below:
Code:
Private Sub submitButton_Click(ByVal sender As System.Object, ByVal e As System.EventArgs) Handles submitButton.Click
Dim trackname As String
[Code].....
View 1 Replies
Jun 4, 2009
I am suing VB 2008 express edition.
I am trying to solve this problem since many days. The problem seems to be rather mathematical, but I faced difficulties dealing with Grids and how to store them using visual basic.
[URL]
In my problem, I have a world of 200 x 200 units divided into 4 x 4 grids (Each of 50 units).
There is a line crosses some areas has the coordinates of Point A (Starting Point) of (XA = 12.43, BA = 159.48) and Point B coordinates (Ending Point) of (XB = 112.95, YB = 20.95).
The question is, how can find the length of (A to C1), (C1 to C2), (C2 to C3), (C3 to C4), (C4 to C5) and finally (C5 to B)?
In addition, how can I indicate the square in which the line is inside?
note that the method should work for all conditions (-ve and +ve line slopes) and for different A and coordinates.
View 1 Replies
Apr 19, 2011
A
/n
/n
/n
.
.
B
Now there can be n no of lines between a and b have to find such content and remove A and B
View 1 Replies
Mar 21, 2009
1. i added a datagrid at runtime on a form. i want to bind a datasource to the datagrid in a button's click event handler, but the datagrid doesn't seem to be visible anywhere else outside the form load event where it is declared. how can i find the datagrid control?
2. and another beginner question.. how to i declare a string that spans multipe lines? i tried with the + operator like in c#, but it doesn't work.
View 3 Replies
Jun 9, 2011
I want to write a program to do Markov chain, but my states are quite large. First of all I calculate all the transition probabilities and revenues for all states(1381860 total states), and store in a multidimensional array. Public RevArr(0 To 9, 0 To 750, 0 To 282) As Long
After that the iteration of markov chain should use these as inputs to calculate the steady-state probabilities. But when I try to run the main code I got this error.Exception of type 'System.OutOfMemoryException' was thrown.
The following is the declaration of second array I add just another dimension for storing all the iterations, but I get this error. Dim stateprob(IT + 1, 0 To 9, 0 To 750, 0 To 282) As single
View 1 Replies
Feb 5, 2011
I have an SQL database, and 50 text files, and Visual Basic 2010 Premimum,I need to find a specific line of text in the text files and then take the next 37 lines of text and save them in my database. I need advice as to point me in the right direction
View 2 Replies
Sep 3, 2009
how to count the lines in textbox1.text (with multiplelines). I don't need to count each character. I only need to count each line from top to bottom in textbox1.text (multiplelines) and I will use Label1 to display the numbers.
example:
1-pauljaones
2-tommyperrry
3-marktoms
4-Jonessmith
5-paulwhite
Obviously this is 5 lines and that's what I need to count.
View 12 Replies
Jun 22, 2010
I have written a simple script to get a bunch of lines from a text file (they will be filenames eventually) which are split by new lines and puts each one into an array..
Dim ary() As String
Private Sub Button1_Click(ByVal sender As System.Object, ByVal e As System.EventArgs) Handles Button1.Click
If My.Computer.FileSystem.FileExists("C:MenuFiles.txt") Then
[code]....
The only thing I can do for now is either use the substring function to remove the first character from all array values after 0 but I don't like that because it's messy and what if the split "works" as I want it to one one of the lines and knocks of the first character when I don't want it to.
View 1 Replies
Nov 11, 2011
I'm trying to the contents of a datatable to a datatable thats allready in my form using the openfiledialog.[code]...
View 4 Replies
Apr 12, 2010
I'm trying to create a program that will merge two data tables (one from a database on a network drive, the other is found in a database on the local machine). Each datatable is identical in structure, only (maybe) differs in content. The structure has a primary key, dateCreated, and dateLastEdited fields (plus other fields). I would like to merge down from the network to the local database. If the primary key does not exist on the local db but does on the network, copy from the network db to the local table. If it does exist, check the dateLastEdited column and copy the latest row to the local table.
How would i set up the code to do this? Any assistance would be great.
View 6 Replies
Jan 23, 2012
How to intersect two dataTables in vb.net 2.0? I want to get common rows in two datatables and add it in third datatable.
View 1 Replies
May 20, 2012
I need to marge two datatables with condition. I have a datatable where the data comes from a local XML Database and another datatable where the data comes from a remote SQL Server. If any update made in the remote datatable I need to update/merge with the local datatable. Here is what I have so far:
Public Sub MargeTwoTable()
Dim SQL As String = ""
Dim RemoteTable As New DataTable
Dim LocalTable As DataTable
[code]....
In this code data comes from the remote database which updates a date getter then the local database . Both tables have "ID" as the primary key. The code is working well, but the problem is that when more than 1000 records are updated this function takes too long using loops.
View 1 Replies
May 28, 2012
I have two datatables which I intends to bind them to Gridview. But I need bit of the first dt and bit from second dt.
Second table has only one row that represents to every row in the same column in first table.
There is no primary key or relationship between two tables though.
For example,
I tried datatable.merge though. Didn't produce the last table i want.
View 2 Replies
Jun 7, 2012
I have a Main Table that contains all the distinct values in a DataTable.DataColumn. also have a Child Table that contains distinct values created by the user.I want to merge the Child Table DataRows that match on the Value column with theMain Table DataRows to create the New Table (shown below).I want to preserve the values in the Selected column of the Child Table.Here is my problem, if I use the DataTable.Merge method the DataRow that contains "888" in the Child Table is added to the New Table, but it doesn't contain a RowState = Added, thus I can't filter for added rows to remove them.
View 4 Replies
Jun 10, 2010
I am using VB Express 2008 and SQL Server Express 2008.I have one DataGridView on a Windows Form, which I am attempting to populate from two datatables. Both tables are identical in structure, but with data from two different dates. The respective SELECT statements are identical, except for the date; one calls for yesterday, one for today. I need a seperate Sum column from each table.I've tried to JOIN, UNION, and MERGE the tables, but that gives me all the data consecutively and I saw no way to seperate the days. I'm able to add DataColumns, and sum the fields with an expression, but is it possible to create DGV columns and bind them to the seperate DataTable columns? [code]
View 6 Replies
