Data Modelling Techniques To Handle Lots Of Data?
Apr 6, 2011
So I have a relatively small website with growing amounts of data - the fastest growing table is pageviews, as we do all our analytics ourselves. The website is made using VB.Net, SQL Server 2008 and jQuery. I am limited to .net 2.5.I currently handle all the data and validation using objects with a list of attributes. Each attribute is of a specific datatype and has a validation function when a new value is given to it. If the data does not validate then it won't even get as far as updating the database and feeds back an error to the user. Currently I have "Manager" classes and "Item" classes - Items are managed by a particular Manager class, for example Users are managed by the UserManager, so to get all the users I just have to call UserManager.Items. These Manager classes are stored in the cache and are created as needed - but once a Manager class is referenced it creates all of the Items within it. This is getting rather slow now, as the company is growing fast. It takes 2 - 3 seconds to load the first page (the page which makes most first-time requests to data) but I am not happy with this time. After that it is nearly instant.
Now the way I see it I have 3 options to speed the website up: Keep it as it is, but try to make it more efficient through other techniques;Change to creating all these objects on every page load, in the hope that it is simply so slow because I am using the cache to store a fair amount of data (about 10 MB or so at least);Forgetting using object oriented techniques and just use straight SQL queries and loop through records.
View 1 Replies
ADVERTISEMENT
Jan 31, 2011
My tile-based map editor project. So far, it works really awesome. Here, look at a picture: [URL] And the screenshot is outdated: Now I got the Pencil, Rectangle and FloodFill tools too!, the tilesets are from that popular (well, not so popular) game engine called RPGMaker.
Currently, this is how the map is structured:
One ArrayList represents the X coordinates.
One ArrayList in each X represents the Y coordinates.
One ArrayList in each X,Y represents the tile.
The tile is an ArrayList containing three elements:
The Image name (Tileset01 in the screenshot), the X coordinate in the image, and the Y coordinate in the image. Those coordinates are from the TILESET01 image, not map.
Getting the idea?
How do you locate a tile in the map when it is clicked?
First I get the cursor's position on the map picturebox, then convert it to a multiple of 32 (the tiles are 32x32, yes), divide the position X and Y by 32. The results are the indexes I need to get to the tile I clicked.
That method works just fine for me.
When the map picturebox invalidates and needs painting... When that happens, I iterate through each X and Y arrayList. When I get the arrayList that represents the tile, I use it to draw on the map.
For example:
Iterating X arrayList.... step 8......
Iterating Y arrayList..... step 3.......
Found arrayList with three elemetns: ("Tileset01",32,96)
Draw image on the map picturebox, source is "Tileset01" on rectangle (32,96,32,32)... draw it on map picturebox's point (8 * 32) and (3 * 32)
And so on.
Now you understand how I am working with my map editor, how I am locating clicked tiles, and how I am drawing it. Everything goes fine, except for one problem: At some point, the user might want to make a 500x500 map.
That means, an arrayList with 500 elements representing the X tiles. Then an arrayList with 500 elements for each X tile, representing the Y tiles. Then an arrayList in each X,Y pair.
Then multiply that result by 5, because there will be 5 drawing layers. How am I supposed to handle such huge maps?! My app explodes when I set the map dimensions to 500x500! 500x500 is the max setting allowed in my app. I also know that 500x500 map editors are possible, because I have seen others before. With great performance.
View 1 Replies
Jul 18, 2011
I have the following "contact" data (exampled below) stored in an xml file.I've created a frmContacts windows form and placed a list box control inside it.I'm new to the list box but plan to have the users manage the list by letting them add, delete, and pick from it. For searching there should also be a sort function/button (by last name then first name).If the user deletes from the list the corresponding data in the file will be deleted.If the user picks from or adds to the list, I'd like to find the best way to populate the corresponding data elements on another form (i.e., in their respective text boxes). This will include the ability to edit/save from this other form too.
<?xml version="1.0" encoding="utf-8"?>
<Rows>
<Row>[code]......
View 1 Replies
Feb 7, 2012
I have an App. I have a lots of datas to handle.So,I wonder if I can click a button and launch Excel and export my datas to Excel.So I can edit my datas in the Excel rather than in my App.
View 6 Replies
Nov 4, 2011
I have a gridview in windows form. When lots of data are loaded from the database to the gridview, my application hang until all the data are loaded to the gridview. Is there any way to solve this problem? I also want to display a gif image [loading.gif] or progressbar when data is being loading to the gridview to show the user that data is being loading to the gridview. But the image and progressbar also hang.
I have also one ToolTip, I want to display that tooltip in the center of the screen. My code is as follow but it is not positioning that tooltip in the center.
Dim x As Integer = TryCastInteger(Screen.PrimaryScreen.WorkingArea.Width / 2)
Dim y As Integer = TryCastInteger(Screen.PrimaryScreen.WorkingArea.Height /2)
ToolTip1.Show("It will take few seconds to display the data.", KryptonPanel1, x, y)
View 2 Replies
Mar 14, 2011
how to change the vb.net code to recieve data from serial port. The packect size of data is 23 bytes so the data recevied event is fored after 23 byte. the data from other end is transfering at 10 ms. after recieving data i have to extract data, add it to database & display the extarcted packet in datagridview.the problem is that i m not even handle the data at 100ms properly. it shows 9,10,6,7 packets in 1 sec. and some time thread synchronization error. i am attaching 2 project the vb6 project is transmitting data at 10/100 ms & the vb.net2008 appliction is doing all the stuff.
View 1 Replies
Sep 7, 2010
I tend to use
If Not IsDBNull(dr("data")) Then
myData = dr("data")
End If
to catch nulls. Is there a better way? It seems I use this over and over again? Anyone write a function to check this something like
mydata = IfNotNull("data")
I don't know how to handle the different data types that could be returned.
View 1 Replies
Jul 9, 2009
how to handle the xml data sent from HTTPService request. simply how to receive the xml data, and read the xml data, then insert the xml data into database. I show you my code:
httpservice in flex:
var xmlFile:XML = new XML(createXML) ;//createXML is a xml String,
covert to real xml file
// testing.text=xmlFile; //it works fine.
[Code].....
View 4 Replies
Dec 16, 2009
I have a small databse program that reads a .csv or text file.
my code
Public Sub OpenToolStripMenuItem_Click(ByVal sender As System.Object, ByVal e As System.EventArgs) Handles OpenToolStripMenuItem.Click
OpenFileDialog1.InitialDirectory = "c:"
OpenFileDialog1.Filter = "CSV Files (*.csv)|*.csv|Text Files (*.txt)|*.txt"
Me.OpenFileDialog1.FileName = ""
[Code]...
i am trying to Catch a data mismatch, if the user happens to select a .csv or text file but they arent of the right type of data. Dim result As DialogResult = MessageBox.Show("Wrong Data Type", "Data Error", MessageBoxButtons.OK, MessageBoxIcon.Stop)If result = Windows.Forms.DialogResult.OK Then
(not sure what to put here)
End If
Catch
End Try
I want to make it so that if the user selects the wrong file, it stops executing shows the message box , the user is only given the option to hit "OK"then it starts the openfiledialog click event again or stops and the user can open the filedialog again It seems that even though the ex is caught the rest of the code will keep running and from what I can tell from the documentation thats by design?
View 11 Replies
Feb 16, 2011
I have a data grid view with columns created when it was filled from a sql table. some of the data types are "bit" (meaning 0 or 1 and automatically shows a check box in the column of in the data grid view).I need to get a handle of the cell that has a check box in it so that when a check box in one column is checked or unchecked, i can change the status of the rest of the columns (only one column can be checked at a time) and then update the entire row in the sql server..when column 3 in the data grid view is checked, app will uncheck column 2,4,5 run sql statement to update changes?
View 8 Replies
Jan 19, 2009
I'm wondering what a best practice is for handling this.I've got a standard vendor table - with VendorId (which is PK, numeric) and a field the users call SimpleId (which is usually a name - don't let the column name of SimpleId make you think it's numeric!).I've got one datatable that comes from a data adapter that looks like this. And this already changes the table in the SQL database...SqlDataAdapter("Select VendorId, SimpleId From Vendor_T", _connection)I've got this datatable on one panel of my form with a binding navigator - and you can change the SimpleId for a vendor with this.
View 16 Replies
Jan 4, 2009
I am new to VB and I have a database that has three related tables. These tables show information about computer engineers.
table one is engineers details
table two is engineers allocated
table three is jobs
My form needs to have buttons where a user can obtain information
View details of all Engineers
View Names of all the Engineers working on a specified job.
Add a new engineer
I have set up the connection to the data base using the wizards and also made the dataset. How to handle retrieving the data from multiple tables.
Here is a screen shot of my data set
View 3 Replies
May 10, 2010
Is this class normally slow or maybe there is something in my process making it slow.I have to import 170,000 rows of data from a fixed length text file into a SQL table. It takes about 2 minutes in a DTS package on SQL server 2000. But we are migrating to a SQL 2008 server and since this code has several manual steps and does not manipulate the data I thought it would be easier to use the TextFieldParser object to handle reading in the data.
Here is the code:
CODE:
View 15 Replies
Sep 3, 2010
I've looked through a lot of snippets online and I see a lot of things similar to
Dim bytes(tcpClient.ReceiveBufferSize) As Byte
networkStream.Read(bytes, 0, CInt(tcpClient.ReceiveBufferSize))
but what I am unsure of when to call such a thing? How do I know when there is data to receive...in visual basic 6 with winsock, there was a specific routine which was called upon data arrival, but I see nothing of the sort for vbNet.
View 2 Replies
Sep 24, 2009
How do I handle user submitted data to ensure best practice programming? I would like to know what are the best practice programming tasks in relation to users submitting data through a web form to a website. I am particularly interested in any C# or VB.NET commands that should be used through out the process from the moment the user hits the submit button until the data hits the database. I have been reading about reasons why you may want to take precautions such as SQL injections etc.
View 4 Replies
Mar 22, 2012
I'm trying to handle DBNull exception while reading data from database. It's my code:
...
Dim SQLRDAs SqlDataReader
...
val1= GetStringFromDB(Trim(SQLRD("Name")))
val2= GetStringFromDB(Trim(SQLRD("Level")))
[Code]...
But still I get Conversion from type 'DBNull' to type 'String' is not valid. error.
View 4 Replies
Apr 13, 2009
I am new to Viisual Basic 2008 and looking for some help or suggestions on what is the best way to handle a spreadsheet style data entery in an application.What is the best way to have data entry on a form that is in the style of a spreadsheet?I have tried it with the Gridview but I do not care for the interface style.I would like to be able to multiple columns under one heading, allow for tabbing from entry field to entry field on a row and have the ability to sort by different columns,to allow for calculations by fiedls in a row (havoing multiple calculations, etc.Is there something in VB 2005/2008 that will allow that or is there a third party add-on that is recommended.
View 6 Replies
Feb 27, 2012
Say I have a 5 GB file. I want to split it in the following way.
First 100 MB is on the file
The rest go some reserve file
I do not want to use readalllines kind of function because it's too slow for large files.
I do not want to read the whole file to the memory. I want the program to handle only a medium chunk of data at a time.
View 1 Replies
Apr 1, 2009
I am building a chat client/server. And I can connect and chat just fine, but when a user disconnects, the server does not recognize this and acts weird, often spamming the last message that was sent by the user..Also, it's like the client does not disconnect from the server even when I close the client app. Once I exit the program, VB acts like it is still debugging. Once the server is shut down however, I get this error:
IOException was unhandled:Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host.
serverStream.Read(inStream, 0, buffSize)I don't know how to make my client and server accept the fact that a client has disconnected...
View 16 Replies
Mar 24, 2011
I have a problem reading data from an RFID-reader. I connect to the reader by tcp and wait for DataAvailable to be true, then reading the data until I got a end of data character. Then I just go back and waiting for a new DataAvailable. This is done in a own thread for the function.
There seems to be some kind of timeout, If I don't got any data in a couple of minutes, then it just sits there in the do/loop waiting for DataAvailable. I hold the card to the RFID reader, it beeps, but there is no data available. I don't get any exeptions, and the information says that the clientsocket is still connected. Are there anything more I can check?
If I put the card to the reader in a minute-interval it seems as this will never occur. So 2-3 minutes idle:ing seems to do this.
Here are my code to read data from the socket, I have taken away some irrelevant code:
Sub test(ByVal ip As String, ByVal port As Integer)
' this sub is meant to run forever
[CODE]......................
View 1 Replies
Sep 3, 2008
Im working on a HSD 32 bit data bus to serial converter via a microcontroller. Now first I want to make a sort of data sniffer to read the data bus (32Bit) .All on the controller side is working and the stream of bytes are coming in via RS232 as 4 bytes in HEX like FD,A5,45,8B . I already converted the bytes in Dec and ASCII to determine what is passed true the bus and how fast and stuff.. Now that is all correct and working very well if the LSB (least significant bit) of the 32 data bus is Data line 0 and the MSB (most significant bit) is Data line 31 but if that is not the case I need to invert the 32Bits (like as I would invert the connector on the bus) . How do I best do this using the 4 bytes I send from the micro controller.
Example:
(MSB) 01001000 01100101 01101100 01110000 (LSB)
HEX= 48 65 6C 70
DEC= 72,101,108,112
ASCII= H E L P
I need to invert it with a button to:
(MSB) 00001110 00110110 10100110 00010010 (LSB)
HEX= 0E 36 A6 12
DEC= 14,54,166,18
ASCII= (SO) 6 : (DC2)
View 3 Replies
Jan 14, 2009
I've got a large and complex database system that is about to get even more complex, and I'm having trouble coming up with a reasonable way to model the new requirements so as to minimize the performance hit.For simplicity's sake, I'm going to use the example of certification exams, although it's not a perfect analogy for my system. So we have the following tables:
- Person
- Certification
- Exam
- Certification_Exams (join table indicating which exams are required per cert)
- Person_Exams (join table indicating which exams a person has taken)
Given that model, we can easily show that Certification A requires Exam 1, and Certification B requires Exam 2 and Exam 3.We can also determine whether a person is certified by checking Person_Exams to see if they have passed all exams required for the certification we're querying. (Note that we are not directly storing certifications for a person; these must be derived. That's one of the areas where the cert exam analogy doesn't quite match my real-life scenario, so please accept that there's a good business reason for it.In my world, the cert requirements are going to change frequently, so a person might get suddenly "un-certified.")Now enter the new requirement: We now need to be able to indicate that Certification A could be satisfied by either Exam 1 or Exam 4.And furthermore, Certification B always requires Exam 2 but also requires either Exam 3 or the combination of Exams 5, 6, and 7.I've come up with a couple different approaches so far, but the subsequent queries to be able to tell whether someone meets the criteria for a given certification are then incredibly complex.
View 11 Replies
Sep 19, 2010
I need to control a device with a GPIB port. I have the commands that I need to use to do this; but I don't know which is the sintaxis in Visual Basic.
For example if I need to send a instruction like "START:0.01" to the external device which are the libraries that I have to include?, how should I write the programming line to do this?
The main tasks that I need to program are sending data, receiving data, and saving data
View 5 Replies
Apr 19, 2011
I have a basic application that allows pre-defined users to use the application.The Username & passwords along with some other basic user info is stored in a simple access dbIn my application I want a user to login, Username & PasswordI create a connection string, a data adapter and a dataset, I then need to validate the password the user has entered.Would it be best to query the access db directly, or the dataset?
View 5 Replies
Sep 11, 2010
I am having an issue on the listbox when the new data value equal to previous data then = to "0". However, I want if the new data value( A ) is equal to the previous data( B ) then ( A ) equal to ( B ).
Delegate Sub ListBoxUpdater(ByRef listbox As ListBox, ByRef str As Object)
Private Sub OnQuoteUpdate(ByRef QuoteUpdate As LingLib.structQuoteUpdate)
UpdateListBox(lbBox, QuoteUpdate.tQuote)
End Sub
[code]....
My problem is when the new data comes out from the API, if it is equal to the previous data and it will appear 0. But what i want is the new data comes out is 0 and it will equal to previous data. e.g: first data comes out is 12.00, in above case, if the second data comes out is 12.00 but it will shows 0 which means unchanged. But in my case now, i want if the first data come out is 12.00, then if the next data is unchanged (0) and it will appear 12.00 and so on.
View 6 Replies
Sep 7, 2010
I am currently working on a API. I am having an issue on the listbox when the new data value equal to previous data then = to "0". However, I want if the new data value( A ) is equal to the previous data( B ) then ( A ) equal to ( B ).
Private Sub UpdateListBox(ByRef listbox As ListBox, ByRef str As Object)
If (listbox.InvokeRequired) Then
Me.BeginInvoke(New ListBoxUpdater(AddressOf UpdateListBox), listbox, str)
[code]....
View 9 Replies
Aug 18, 2011
I just know it works. So as a personal project I'm trying to learn in VB/C# all about Cryptographic methods and how to use those techniques to encrypt and decrypt files. [code]
View 2 Replies
Jan 22, 2011
i am new to asp.net and i was told that i need to know UML thoroughly to build successful software, is this correct? i mean cant i just "code and fix" and "model" in my brain. how important is UML and what is the best way to learn it?
View 3 Replies
May 19, 2011
From what I understand, it is still impossible to prevent software piracy until a global Internet is established.
In the meantime, I'm looking for suggestions on how to protect any new projects I design.
View 9 Replies
Apr 10, 2009
Ok heres my idea Start(button)> start timer1 countdown> keypress> start timer2 countdown> keypress2> if keypress2 happend twice> then keypress3> loop> end>I understand the science behind it but I fail to grasp the knowhow of the program lang if anyone knows how this could be done in VB Express.
View 9 Replies
