Data Validation On Huge Flat-file?
Jun 5, 2011
I have a huge amount of data in flat-file format that I would like to validate with Visual Basic 2010. Basically the data is a text file with 300 comma separated fields and 500,000 records.
Any tips, sample code or examples? Are any .NET classes specifically geared for data validation?
As I have different validation rules for each field (some fields are dates, some are currency, etc.) I would like to store the rules away from the code (e.g. in an SQL database table). Has anyone seen this done before?
View 1 Replies
ADVERTISEMENT
Oct 15, 2009
Our application is using VBSQL.OCX on Sql server 2000 to create temporary tables and load pipe delimited data from a text file with predefined format into the tables. It is failing intermittently doing this. 99% times it works perfectly. But on occasions it does fail with error. Once the process is rerun with the same input file it runs successfully.Basically the architecture is of Store/Head Office where there are around 200 remote stores sending set of data files to the Head Office. At HO our application sequentially processes each store which has all the set of files received, creates a corresponding table, loads the data from the file into the table and moves onto the next store.If it is for some reason not able to load the file for a store, it gives a failure message and continues with next store.Error which the OCX throws is "Attempt to convert data stopped by syntax error in source field"
View 1 Replies
Feb 25, 2010
i have been searching the web for days and yet to find a clue on how to extract specific data from a flat file, i know how to extract all the data from flat file but not "specific data".
my problem now is that i need to extract a certain data out from a very very messy log file, what i need is to extract the decimal data right after "score=". it is in this format "score=1.938249".
for the decimal data right after "score", it is very random, there is no fix length to this. and also "score" does not fix in a specific position, it could be starting of the string or ending of the string of messy data.
just a peak on my messy data,
"time 20:34 17/1/2010 vlog=0 C:d7ad45f396850c532ef0ccf0da48d80di386peisgh437 ipint=98.1857991 score=1.78495 event@&1.13845"
"time 20:34 17/1/2010 vlog=1 C:Documents and SettingsDefault UserDesktop score=4.085651123 ipint=70.1578356 event@&0"
View 9 Replies
Mar 4, 2011
My app is a fairly simple and im sure with the right knowledge this would take 2 minutes.. but sadly iv been at it for hours. So my scenario here is a huge network with 20+ file servers all with data spread out across them it can take hours to find the folder required.
Hence this app.. so it consist of a input box, progress bar and a button, using FileSystemObject I plan to interrogate a directory to see if a folder exists (folder specified by user input) however it need to check multiple directories, in my old app I used a list box which wrote to a text file allowing the user to add additional directories then using if statements proceeded through the servers.. however they have added that much that my IF's have ran out really because i know it should have been a loop
Imports System
Imports System.IO
Public Class Frm_main
Private Const FILE_NAME As String = "C:directories.txt"
Dim objReader As New System.IO.StreamReader(FILE_NAME)
Dim Lines() As String = IO.File.ReadAllLines(FILE_NAME)
Dim Dir As String
[CODE]...................
View 1 Replies
Oct 25, 2011
I'm trying to convert a DTS package to an SSIS Package. My package contains a Data Flow task that pumps data from an OLE DB connection to a Flat File destination.The data in the flat file needs to be arranged in a certain way (eg: spacing), So how do I do that? Is there any data trasformation tool that will help me achieve this? If so how would it be done?THE DTS CODE (to put things in perspective)
sLineItem = Space(10)
sLineItem = sLineItem & Space(10)
sLineItem = sLineItem & Space(10)[code].....
View 1 Replies
Nov 16, 2010
im using vs2010 and SQL server 2005. what is the best and fastest way to load large volume of data into treeview
View 6 Replies
Apr 6, 2011
I have 2 representations of similar data from 2 different systems and I need to match each entity in one system with the entities in the other.[code]...
View 1 Replies
Oct 14, 2010
I have code in an ASP.NET form that needs to, depending on user entry create messages in the database. We are speaking of potentially thousands of db entries. How do I protect against deadlocks, I mean apart from using Transactions and setting IsolationLevel to Serializable, as well as using WITH(NOLOCK) statement on my select statements since I don't mind a dirty read.
[Code]...
View 1 Replies
May 19, 2012
web method:
[Code]...
with this condition(PACK_DATE > "1388/11/07" 366 records) everything works well.but when i change date to 1388/11/06 1260 records or 1388/11/05 5460 records or ... following error occurs:
[Code]...
View 1 Replies
Jun 15, 2010
I have a huge .txt file which contains X & Y data seperated by serialization. Each serialization contains 1K x and y data points, the text file is about lines 1,340,000 long. Sometimes I need to go into the .txt file and find certain sets of data so I can plot and overlay them which gets tedious. I am going to write an application that I can just write the serialization into a text box on a form and and it will seach all of the txt files for the right data, put them into arrays, and then load them up into excel. My question is how would I search the .txt file for the correct serialization? Would I first read it into an array and then try and split it that way, or is there a way to just open the .txt file and search for the serialization without putting it into an array first?
View 3 Replies
Jul 1, 2009
I've gotten the code I posted a few weeks ago working to export a tab delimited file. ([URL]) Now, I've already converted it from tab delimited to "tilde delimited." My goal is to import this flat file created as an EDI 850 Purchase Order. My ordering system rejects the file, yet it looks identical to other working files. My thought is the encoding / format is wrong.
[Code]...
View 1 Replies
Aug 9, 2010
Such as the text file has 5000 lines and 500000 columns. Each line has the format "xxx,yyy,zzz,000,...."
View 39 Replies
Nov 24, 2010
I have a huge text file that stores a huge matrix. The size of the file has 120GB. There are tremendous rows and 25 columns. I am only interesting a coulple of columns, say column 3, 6, 8, 13, 20. How to create a small matrix which contains these columns. I want to find a effective way rather than read the entire file.
View 1 Replies
Jan 6, 2010
Using SQLDMO you could export a file to a flat file however this functionality is not available in SMO. From what I've read you need to use SSIS to accomplish this now. how to do this in VB.NET or C#?
View 4 Replies
Mar 9, 2012
I need to generate a flat file from my scanner MC9090 or MC9060 to my computer.
View 1 Replies
Apr 22, 2009
Is there a way to import data from a flat file or a .txt file to a sql server table and implementing the process from vb.net??
View 1 Replies
Oct 3, 2009
Im trying to tell my button to go directly to a website, it works fine when i path it to the web browser itself, but then when i add it to go to the sppecific site it says system cannot specify file path. Heres the code Rob gave me and it didnt work ""[URL]""" Heres my exact line: Process.Start("C:Program FilesFlockFlock.exe ""[URL]"")
View 5 Replies
Oct 23, 2009
I am in process of designing an architecture for an application. Following is the high level requirement.
1. The basic requirement is to convert a flat file into XML
2. The input flat file can have various formats, the delimiters, the data positions may vary.
3. The output XML format is predefined. There is a specification defined in a word format, which I think I need to define it as an XSL or XSD.
I am planning to follow some design pattern. Not decided yet. The architecture should be flexible to extend the input file formats and accommodate any changes in output specification.
View 2 Replies
Nov 19, 2009
What is the best method to parse through a Comma Delimited .dat file? I am using VS2002 and VB. My code will be looking for certain placeholders within the files and pulling the value from that element to populate a table in SQL2000.Here is a sample of one row that is in the database so far:17768 01 1770 003 2009-11-01 00:00:00 5065.14 NULL NULLEach spaced section is taken from a part of the .dat file, but being manually keyed in for now. My job is to automate the reading of the file, and writing it to the SQL database.Here is a sample of the flat file I am trying to parse:
0001,00003470,091411," "
0002,1000,0037707839,0000304220,0000282256,00000387
0003,1000,006795,0004,00000000,0000,00000000,0000,00000000,0000
There are about 12-13 'records' in each flat file, each row being a record as the sample shows.
View 1 Replies
Sep 19, 2009
Is there anyway to use Microsoft File Transfer Manager as an OCX or DLL to manage huge downloads? I'm planning an app.
View 1 Replies
Jun 2, 2011
I am trying to create a datagrid and export the contents to a text file using VB.NET and I am doing this inside an SSIS script task in order to automate the process to export a dynamic table to text file. I don't get any error and the files are created but the files are empty. [code]
View 2 Replies
Jul 12, 2011
so i have a file that i need to read in vb.net. Each line is 32 chars long and each line consists of multiple fields. Meaning from char 1 to 9 it idicates ID then char 10 indicates an identifier,11-14 indicates a value field. I've looked at Readblock and it's not the tool for the job. Is there a way to break up each line into its respective fields other than just reading the whole line and breaking it up as a string?
View 3 Replies
Dec 8, 2010
I place these 2 questions in this forum simply 'cause those that know GDI+ best are probably .Net vs. VB6 classic users... So here it goes.I'm using GDI+ flat file to generate a textured brush with image attributes. Creating these are no problem, but I can't find documentation that says whether disposing of the brush will dispose the original image & attributes handles I've assigned to the brush or whether I need to destroy those also? I can't locate anything that says whether GDI+ copies the handles or uses the ones I provided. Destroying the handles well before the brush is destroyed does no damage. Thus my assumption is that the handles are copied. But you know what they say about assumptions.
2. Likewise I use a GDI+ function to return the image assigned to a textured brush. Again, I don't know if that returned image handle is to be disposed by me or not. The returned handle changes every time I call the function, so I'm assuming that I should be destroying it. Thanx to all that can shed any light whether from personal experience or pointing me to some black & white documentation.
View 1 Replies
Mar 21, 2011
I have to read in several thousand lines of a EDI 4010 or 5010 formatted file into a database. This is medical claims data. In my current process I read each row of the file and parse it into a string array. I then assign each value of the array to a string variable and when I detect a new claim header or detail record I write the variables to the SQL server in a SQL insert statement. Example below
[Code]...
View 3 Replies
Dec 29, 2010
I am looking at Visual Basic 2008 Step By Step. I need to read a flat file and insert the records into my SQL database. My code so far:
[Code]...
The INPUT line needs to be fixed. From what I can see, I need to move it all to a single field and use a split to parse it, correct? As I look at the entries on this site, all the code seems to use stream readers. Why would the book not do that? Am I missing something?
View 6 Replies
Dec 9, 2009
The following is my connectionstring: server=(local);database=JnJ3;uid=sa;pwd=sa May I know how can I avoid timeout expired because I keep on receiving error message when deal with huge amount of data. The timeout period elapsed prior to completion of the operation or the server is not responding.
View 1 Replies
Aug 11, 2010
I have dragged across the details of a table from the data sources onto a form. So VS2008 automatically generate the BindingNavigator, DataAdapter,Dataset, ...etc. On the form I hv also placed an errorProvider and listbox, which is bound to the bindingsource. So as user clicks on the items of the listbox, the values of the textboxes changes accordingly. My question is how do I validate the values in the textboxes.
View 4 Replies
May 4, 2012
This seems like I only need a line or two of code? This is what I need to do:
Add data validation to the new phone field on the Contact Us form. We would only accept phones with the following format: 995-999-9999 Complete the contact us form so that you can submit it as an email. Send all your test emails to blah@boo.net. You can use any email account you have to send the email.
[Code]...
View 1 Replies
Nov 5, 2010
Add data validation to the new phone field on the Contact Us form. Wewould only accept phones with the following format: 995-999-9999Complete the contact us form so that you can submit it as an email. Send all your test emails to blah@boo.net. You can use any email account you have to send the email.
View 4 Replies
Jun 15, 2010
I've built a UserRepository interface to communicate with my LINQ to SQL Data layer, but I'm trying to figure out how to implement validation.
Here is what my AddUser subroutine looks like
Public Sub AddUser(ByVal about As String, ByVal birthdate As DateTime, ByVal openid As String, ByVal regionid As Integer, ByVal website As String) Implements IUserRepository.AddUser
[Code].....
View 2 Replies
