DB/Reporting :: Move Data (400 GB) Into Database Then Be Able To Run Various Algorithms Determining Correlations Of Time Series
Mar 12, 2012
I'm faced with a (fun, actually) data mining problem; I have raw ASCII files from instruments, and I want to move that data (~400 GB) into a database, then be able to run various algorithms determining correlations of time series, etc. I would like to write the mining algorithms in Visual Basic (.net, VS 2010 right now), and be able to do visualizations with VB code I have in hand.On the nature of the data: think of a set of several thousand devices, each recording a measurement at a given interval - so I'm talking time-series vectors. It's not more complex than that - though I may have vectors with holes, etc. - not sure what problems of that sort lurk in the data.I spent today re-acquainting myself with VB.NET's interface to (in one case) a Microsoft Access database. What used to be fairly simple - DAO I think it was - involved tables, recordsets, etc (and that would likely be fine). Now I seem to be required to have a weird variety of generally useless objects, e.g. 'adapters', 'datasets', etc. The problem is that I know exactly what I need, and all this extraneous stuff just gets in the way (certainly in coding complexity and opaqueness, and likely in efficiency as well). If any of these mechanisms gave me a kind of virtual access to the entire dataset, and let me control caching parameters, etc.it might be great, but I found nothing along those lines. It seems like useless bloat, though I suppose it must be useful to someone.Anyway, I tried a number of different approaches, and none seemed at all aimed at what I need to do: efficiently do math on a large dataset. I can't believe I'm the first to have this problem, but I can find no useful wisdom out there. I'd be comfortable with pretty much any underlying database mechanism: MySQL, SQL Server, MS Access, but ideally something generally SQL based (I may eventually have to transition this entire system to draw from a client's SQL database, though that's not an overriding concern now). Other than that I want simplicity and efficiency. I thought my old ODBC techniques would work, and to some extent they do, though modifying tables seemed to have bizarre problems (no errors, but not modifications either).
I do have a fairly aggressive deadline to show some algorithm results, so my focus in the short term is to get something reasonable working *in* the short term - in other words, it's less important to me to pick the 'fastest' relational database than it is to pick a database that lets me focus on coding the algorithms, not working through tedious data access coding. If this db could be any smaller, I'd have tried to do it all 'in memory' at least for proving concepts; I don't want to have to learn an entire jargon and approach just to be able to retrieve data points.Perhaps I'll need to bite the bullet and just write something myself, a .dll perhaps just to save and restore large time series vectors. It seems a bit frightening to me that one would have to do this in this age, what with all the database systems out there, but I don't have much time to work through arcane interface logic.
I want to move my programs and the database to another computer and hard-disks. It looks to me as though I nearly have to re-create the controls and rewrite some of the code to make it so the programs still work after the move. At least with ODBC all I would change was the DSN definitions external to the programs. Is there an easy way to move or rename a database without loads of work?
I have a simple app that just shows the current time for specific locations around the world. I need to have it adjust for daylight savings time. I *THOUGHT* I'd already coded this but....
I thought I would use "Today.IsDaylightSavingTime()" and add 1 hour to the clocks. It isn't working?
If Today.IsDaylightSavingTime() Then HawaiiOffset = HawaiiOffset + 1 PacificOffset = PacificOffset + 1
I have two databases, d1.accdb and d1b.accdb (So I have two connectionStrings). In both of these databases there is a table called Postings with exactly the same fields. All i want to be able to do is query the Postings table in d1.accdb for all the records where the users name is xxx. I then want to move all that data to d1b.accdb. I found another thread that was started in like 2007, and tried to follow their instructions, but cant seem to get it to work. Here is the code I am using:
[code]...
When I run getUserPostings(), i get a OleDbException saying invalid argument. Does anyone know how to fix this? or perhaps know another way to do this?Once i get this to work im guessing i would just add "WHERE UserName = 'xxx'" to the end of the OleDbCommand Strings.
I am using Visual Basic 2008 for a Windows Form application. I have a PictureBox on the form that I need to display a series of 12 JPG files, one every four seconds. For the life of me, I cannot seem to figure out how to get the Timer control and its Click event to work.
In pseudo code, I need something as simple as the following:
1. show JPG #1 in picture box
2. 4 second delay
3. show JPG #2 in picture box
4. 4 second delay
etc.
I have a Timer on the Form set with Interval = 4000. I have a PictureBox. I have all the images in the Project. I am embarrassed to say that I have spent 4-5 hours searching the Web for sample code or something helpful, but am still stuck.
All the elements are there. It's just a matter of understanding how to pause the image displayed in the picturebox for 4 seconds before moving onto the next image in the series.
need to get the data from Cell(2) on my gridview but cant figure it out. id like to put it in a varible im using VB 2010 express ed. what im aiming for is to show a popup box showing of my note and to show the date. by compairing in a loop of all entries if the date is falls on the date that was picked on the entry.
How to i insert Variable @ClassID(If Identity Propeties) @LecturerID , @StudentID and a Date from Calendar Control into the Date Collumn , into a Database Called Fingerprint, TableName= Attendance.
The Collumn Are ClassID, LecturerID, StudentID, Date.
I have an Access database with a table called Events.I created a new VB.net 2008 project and created a MicrosoftReportViewer control. I also added a "Report" component (which created a .rdlc file).I am easily able to drag things from my Access database I added to the data sources onto the report. However, one part of the report I would like to add data to a Matrix that I calculated with VB (ie it is not in the database). The database has a list of names and a bunch of times associated with each name. and then I calculate some fancy stuff based on those times. I would like the matrix to be a list of names (from the database) on the rows, a list of dates (which I specify in the code) on the columns, and the data of the matrix I also calculate.
I would like to return the data in a database query as an array, but the problem is, Im having trouble working out how to do it, here is how I would like the data to be returned:
I could create a multi dimentional array, but the results could range from 2 rows, to 2000 rows, I looked into redim, but you can only redim the last dimention, I would need to the first and second, but not the last.
Ive got a web page so that the users select a day from the calendar and the date is input to a text box, the format is dd/mm/yyyy, it is the same format in the database table.
I want to select the data from the database when it is = the date in the text box but it is retrieving data in mm/dd/yyyy format. My sql is:
SELECT Booking.Date, Booking.RoomNo, Booking.Lesson, [NoComps] - " & students & " AS diff FROM Rooms INNER JOIN Booking ON Rooms.RoomNo = Booking.RoomNo WHERE booking.Date= #" & txtdate.Text & "# AND booking.Lesson = '" & lessonlist.SelectedValue & "' AND booking.Type IS NULL
how can i make sure it retrieves the data using the correct date format?
I'm working on a VB.Net Windows form that is supposed to manipulate an Access Database. I can Already read data from the database (placing them in a combobox) using the following:
Code: Private Sub Form2a_Load(ByVal sender As System.Object, ByVal e As System.EventArgs) Handles MyBase.Load OpenFileDialog1.FileName = "" OpenFileDialog1.Title = "C# Corner Open File Dialog"
I was wondering which way would be the easiest in VS2005 to connect to and pull data from a MS SQL database. I have already setup the database connection by choosing the datasource for a DataGridView but I'm not able to retreive any information.
The steps I have taken so far:
1. Make new form
2. Drag and drop DataGridView onto the form
3. Setup the data source and respective username password
I am using Visual studio 2008 to build a web application. What I am trying to do is retrieve data from a database and populate text boxes based on the results.
I need to extract specific data from a database. With starting point in this data, I will collect information on the internet. After the information is collected I need to check if this information is already stored in the database, and if not, then add it. This action is being repeated about 500-50000 times each time the procedure is being executed. (not often)
The approach I'm taking seemes sleazy and ineffecient and I was hoping you could point out some places where I might be able to optimize it. I've two open connections at the same time. The first I use to extract data, and it stays open throughout the whole session. The second connection is being used first to check if an entry already exist and if not, to add the data.
The reason I am using two connections is due to the fact that I got a reader bound to the first one, so it is rendered unavailable.
I was thinking that maybe checking if the data existed, and adding it if not, maybe could be contracted into one command execution instead of two. I don't know if it would be possible to do this with one connection only as well, as it seemes more proper.
I recently made a database that saves user information into a MS Access database from a form.
Code: Private Sub Button2_Click(ByVal sender As System.Object, ByVal e As System.EventArgs) Handles btnSave.Click objDA = New OleDbDataAdapter("SELECT * FROM CDetails",
[Code]....
Now when they go to the next form, I have labels such as lblFirstName, lblSurName ect. I want to display the information that the user has inputted into these labels. How can I fetch data saved into the database and display it onto these labels? and how can I select rows?
i am trying to update database with updated data in data adapter. here is error message. "Update requires a valid UpdateCommand when passed DataRow collection with modified rows." here is my code.i get this error when i try to update dataadapter with following line.
I've been staring at this and can't seem to come up with a solution. I've tried several different methods that i found on the web but to no avail. I'm hoping you all could take a look at the code below and give me some direction as to why I'm receiving the error "The INSERT INTO statement contains the following unknown field name: 'Deposit_Date' . Make sure you have typed the name correctly and try the operation again. Sorry if it's kinda jumbled below
Code:
Dim SQL As String = "INSERT INTO Deposits ( Deposit_Date, Deposit_Time, Location, Miscellaneous, " & _ "Miscellaneous_Adjustment, Dispense, Dispense_Adjustment, " & _ "Replacement_Card, Replacement_Card_Adjustment, Deposit, " & _
I am using the following Select Statement to retrive data from my .csv file database:
Code: Dim comm As New OleDb.OleDbCommand("Select *, Left([Street Full Address], IIF(InStr(12,[Street Full Address],'-')>0, InStr([Street Full Address],'-')-1, Len([Street Full Address]))) as StreetFullAddress From " &
I am creating a reporting app that pulls information off of a Sql database and populates the data in a datagridview control (DVG). I am using tab pages (Tabcontrol), and I have about 10 tabs with within the control. What I have done is created a dynamic control that will populate in each tab page once the tab is selected. I am using a split container and in my split container.panel 1 is where all my buttons and search functionality will be. In my split container.panel2 to is where the main DGV control will be. I have got this to function properly, however now I am unable to add anything to my split container.panel1. I would like to add diffrent labels and buttons in the panel1 one without putting another panel on top of the split container.panel1, and also if i do this when I resize my container the panel will just sit there. I am using the selected index changed event to get my split container to show in each of the diffrent tabs
Private Sub searchTab_SelectedIndexChanged(ByVal sender As System.Object, ByVal e As System.EventArgs) Handles searchTab.SelectedIndexChanged searchTab.TabPages(searchTab.TabPages.IndexOf(searchTab.SelectedTab)).Controls.Add(splitContainer1)
I'm trying to make the move from VB6 to .Net; I know Bill needs the money, but my real motivation is to move my VB6 client-server app to .Net before Bill finishes killing off VB6. My VB6 app uses SQL Server 2005 Database, ADO, and heavy inline queries and some stored procedures. I'm weak on oop, but willing to go there.
My biggest concern so far is not being able to fully use my database w/o extensive redefining.I think Bill went oop crazy and forgot why we use databases. I wanted to bind all objects in my code, but coding the work of the wizards is not coming easy for me. My qt pertains to using ADO.Net or LINQ for database access. I'm going to have to put in a lot of book hours either way, but which way should I go for a client server app? I read that Bill is handing off the LINQ to the ADO.Net team. Is LINQ the future for client server or is it a dead-end?
I am attempting to determine every time a file is accessed, either opened, changed, deleted, created, or other various times such as a files is accessed like to be sent as an attachment in an email or something of that sort.
I am using VB.net Express I need to produce a chart so am using MSchart object.I am using the following code to try to add data (2) to column 1 of a series
Me.Reject_Chart.Series(0).Points.AddXY(1, 2)
I do not get an error immediately but then before the form can be refreshed I get a stack overflow.how to I set this data?
I like to update data from an access database every 100ms. This works fine, but it does not look good on the screen. It refresh the whole screen on my laptop and it is not looking good. Is it a good way of updating data without getting this disturbanses in the screen?
I want to store date time picker into database .I am using date to search my reservation information and how is the code for search out the information from database.
