Implement Multithreaded Crawler - Add A Unique Url To Each Thread To Go Process
Feb 1, 2010
I would like to implement a mulithtreaded crawler using the single thread crawler code I have now. Basically I read the urls from a text file, take each one and crawl and parse it. I know how thread basics of creating a thread and assigning a process to it but not too sure how to implement in the following way:
I need at least 3 threads and need to assign a url to each thread from a list of urls, and then each needs to go and fetch it and parse it before adding contents to a database. [Code] Now the code maynot make sense but what I need to do is add a unique url to each thread to go process.
View 4 Replies
ADVERTISEMENT
Feb 1, 2010
I would like to implement a mulithtreaded crawler using the single thread crawler code I have now. Basically I read the urls from a text file, take each one and crawl and parse it. I know how thread basics of creating a thread and assigning a process to it but not too sure how to implement in the following way: I need at least 3 threads and need to assign a url to each thread from a list of urls, and then each needs to go and fetch it and parse it before adding contents to a database.
[Code]...
View 4 Replies
Mar 18, 2010
I have written a multithreaded crawler and the process is simply creating threads and having them access a list of urls to crawl. They then access the urls and parse the html content. All this seems to work fine. Now when I need to write to tables in a database is when I experience issues. I have 2 declared arraylists that will contain the content each thread parse. The first arraylist is simply the rss feed links and the other arraylist contains the different posts. I then use a for each loop to iterate one while sequentially incrementing the other and writing to the database. My problem is that each time a new thread accesses one of the lists the content is changed and this affects the iteration. I tried using nested loops but it did not work before and this works fine using a single thread.
Here is my
SyncLock dlock
For Each rsslink As String In finallinks
postlink = finalposts.Item(i)
[CODE]...
Finallinks and finalposts are the two arraylists. I did not include the rest of the code which shows the threads working but this is the essential part where my error occurs which is basically here postlink = finalposts.Item(i) i = i + 1
ERROR: index was out of range. Must be non-negative and less than the size of the collection. Parameter name:index
I tried copying it to a new list but dosent work.
View 9 Replies
Oct 31, 2011
I have a linq query which loads approx. 1000 lines into a variable, during that process I want to display a progressbar, not necessarily stating the percentage, can be marquee style, doesnt matter.
This progressbar is on a modal form to precent the user from interacting with the app for the time the query's running.
[Code]...
View 3 Replies
Sep 14, 2011
I'm developing an Outlook 2007 Add In with Visual Studio 2010 (VB).In the project I have a routine that opens, saves and closes Word 2007 documents.While the routine is executing, if I open Word 2007 application I see all the documents opening and closing, making it impossible for me to work in Word 2007.How can I launch the routine using a unique Word 2007 process?
View 5 Replies
Feb 6, 2011
how do I force a particular set of vb.net codes to run in a new thread or process.? Edit 1: I am trying TTS in vb.net but whenever click the play button , The whole program freezes and I cannot do anything else until the speech is over
View 3 Replies
Jul 24, 2009
i have a link of working crawler sample code written in either c# or in Vb.net
View 3 Replies
Jul 12, 2009
At the moment, I am writing a class that deals with starting an application in a thread, as well as being able to monitor the closure of the thread and process, and the creation of the thread and process through events. I have absoloutly no problems with monitering the creation of the thread and its internal process within the subs that create them. I placed notifications within the "myprocess_Exited",(for the process closure) and "myprocess_Disposed"(for the thread closure) events.
The problem comes when I terminate the outside application manually; these events do not appear to fire when the application is closed by the user, rather than by code.I thought, since the application closes with an error code when it is terminated in such a way, that it raises the "myprocess_ErrorDataRecieved" event. But this doesn't appear to happen either. Here is some of the code from that class:
'Some of the decleratoins
Event ProcessStarted(ByVal Process As Process)
Event ThreadStarted(ByVal Thread As System.Threading.Thread)
[code]....
View 3 Replies
Mar 21, 2011
The code has gone bad. It is in debug mode and jumping around everywhere.
Error = "The process or thread has changed since last step."
Here is the code. I just create the file "start.csv" to initialize the shell command. This is very unstable for some reason. When I run line by line in debug mode it starts jumping around irrationally. It gives me versions of the above error.
Option Strict On
Option Explicit On
Imports System.IO
Module Module1
Public WithEvents Tmr As New Timers.Timer
[Code] .....
View 10 Replies
Mar 24, 2009
Here's the problem I am trying to solve. My program can export a file as a pdf. and other formats. I can open the file using process.start. My file doesnt have a good name so I am using a GUID as a name.
[Code]...
View 2 Replies
Apr 10, 2012
Basically, I am just trying to close the thread/process after it's done.
Private psi As ProcessStartInfo
Private cmd As Process
Public Delegate Sub InvokeWithString(ByVal text As String)
Private Sub Button1_Click(ByVal sender As System.Object, ByVal e As System.EventArgs) Handles Button1.Click
Dim psi As New ProcessStartInfo("m7.bat")
[Code] .....
I am trying to close the thread so once it's done, it closed for good and my form is closed as well. I tried Me.Close() which results in: "Cross-thread operation not valid: Control 'Form1' accessed from a thread other than the thread it was created on." Error.
Me.Invoke(New InvokeWithString(AddressOf CloseME), e) - Results in parameter mismatch error.
However, things like Msgbox("Done") works perfectly.
View 7 Replies
Jan 18, 2010
create a web crawler using vb.net?
View 5 Replies
Oct 21, 2010
I'm attempting to do a school project dealing with a simple web crawler. I have a form with a web browser control embedded that loads a web page will all the available courses. I have a text box and a button design for a user to search for a specific course by using the four letter department abbreviation. The page that I have loaded has all the department abbreviations as hyperlinks.is it possible to search the page using the four letter abbreviation specified by the user and if the search finds a corresponding hyperlink open it. Then I would use a loop to repeat the process of opening each class offered by that department and obtaining information such as course name, section number and so forth.
View 2 Replies
Jul 7, 2011
Is it possible to run a external program/process example notepad in thread of you vb application?
View 2 Replies
Feb 10, 2012
I have two applications. For performing certain functions, i run a second exe, via a process, and wait till it is complete (exited) to continue with my main application. I run through many repetitive functions on different files.Everything runs fine however, when i am doing a run and it is on the x loop number, the main application seems to halt the secondary process exe. I can only assume this is what is happening as, if i close/stop (not pause) my main application in VB, the second exe continues from where it was halted until it completes.
View 4 Replies
Apr 16, 2010
A long time ago, on a PC far, far away from Windows, one MS-DOS program (the father) could run / execute / spawn another MS-DOS program (the son). When the son terminated, it could set a return code which could certainly be interrogated by an MS-DOS batch program or, I think, by any father program that invoked it (depending on the language in which that program was written). I have a small utilility which I want to invoke from my application, and, since that utility sets a return code, I would like to interrogate it in my application when the utility terminates. I've looked through the MSDN Library but could not find anything that seemed to cover this situation. Is there a way to do this in Visual Basic 2008 Express
View 2 Replies
May 14, 2010
I'm using VB.net 2005, .net framework 2.0
I'm currently having a form that has a button starting a long process, with feedback into another windows textbox (call frmProcessLog).
I'm using a Background worker to laucnh the process, upgrade the status of the process and display feedback on the frmProcessLog textbox, and to know when the thread process ended up, (from the cancel button, from an error or normal termination).
the main thread process has some msgbox in it, asking questions to the operator (yes/no/cancel).
I've added the msgboxstyle.systemmodal to the msgbox call which works "ok" excpet if the operator decide to ignore the showing message box and click the cancel button on the frmProcessLog form, which is still clickable even though I supplied "systemModal" to the msgbox. this makes the msgbox hidden behind the frmProcesslog form and the process can still continue.
If I close my frmProcessLog form, I will see the previously msgbox behind it and If I end up click one of the button it will end up at an application error because the thread that called it is not existing anymore.
So I would like that the msgbox to be launched from the thread making it as if it was launched from my frmProcessLog window in a modal way.
View 4 Replies
Jun 28, 2009
It is my previous problem in vb6, its hard to multithread using vb6 so I migrate my codes to vb.Net.How can I multithread this two loops so that this process is running at the same time.Because when I start my second Loop, my first loop stop in executing. It will start again when my 2nd loop finished.
[code]...
I have 2 connected GSM Modem in usb port.That codes will send SMS in all contacts in my ListBox.
View 27 Replies
Feb 13, 2012
I am using VB .Net to access the eBay API and store all completed orders in a database. This database is then fed into a proprietary shipping system, which can not handle an order number larger than 20 characters. eBay returns an order number like so 230407279314-680141236013 which is too long. The order number is always 12 numbers a hyphen and 12 more numbers. What I need to do, is turn this (the result can be alpha numerical) into a shorter, unique order key to store in my database alongside the true orderId (so that this can be referenced by the shipping software instead of the actual order number). The reason for the 20 character limit is the barcode algorithm used. Is there any way to achieve this in VB .Net 2010? This number can be anything unique, so long as it does not exist already (even a good uniqueid function would work, but I would have to query the database to make sure it isn't taken)
View 2 Replies
Jan 10, 2009
I'm using the datagridview. This is the sample data:
[Code]....
How do I parse through the grid to rename the duplicate values in a particular column (in this example it's the l_name field) into such a format?
[Code]....
View 1 Replies
Mar 21, 2011
How generate the unique no. 1,2,3 and so on .... on button click of each new user ..
the code mentioned below is a readwrite coding in vb.net ...
but the problem is it generate the same id for different users on button click event... but i want the no. of times button clicked the new ids will be generated
Protected Sub Button1_Click(ByVal sender As Object, ByVal e As System.EventArgs) Handles Button1.Click
Dim FILE_NAME As String = Server.MapPath("counts.vbi")
[Code].....
View 1 Replies
Aug 26, 2009
i'm trying to build websites crawler and i having a bit of problem creating recursive function to get all the site link, provide a link to an example ?
View 2 Replies
May 8, 2011
Unique constraint - these columns currently doesn't have unique values..?? I have created Unique Constraint with 3 columns. My code works perfectly but once - two users @ same time entered same data and somehow it saved in DB; after that incident. This UniqueConstraint gives me error - These columns currently doesn't have unique values. How can I check if user enters this kind of entry or how can I restrict the Entry?
View 2 Replies
Aug 13, 2010
In one application, I need use 4 simultaneous threads.When the threads finished, I need to update the text of a TextBox.So, I create 4 Textbox, and I wanna the threads change the text to FINISHED.Each thread change one distinct TB.I use this "example"
Dim Terminados As Integer = 0
Private Sub frmEnviarMensagem_Load(ByVal sender As System.Object, ByVal e As System.EventArgs) Handles MyBase.Load
Timer1.Enabled = True
Timer1.Start()
[code]....
The problem is, I have an error in the red lines.The error is: INVALIDOPERATIONEXCEPTION "Cross-Thread operation not valid: Control tbPlaca1 Accessed from a thread other than the thread it was created on."I don't use cross threading. Each TB only was accessed for the correspondent thread, and for the "main program".
View 3 Replies
Nov 3, 2011
This is the error message I am getting:
"Cross-thread operation not valid: Control 'Panel1' accessed from a thread other than the thread it was created on." The reason I am getting this error is because I am opening up a new form and then calling these three things:
Panel1.Show()
Label1.Show()
Label2.Show()
why I get this error message because it doesn't occur normally if I open Form2 after closing Form1, it only occurs when I open Form2 after closing Form4.
View 4 Replies
Jul 21, 2009
I have done a program using vb2005 to display reading from my microcontroller bs2 board but have encountered some problems. My code are as follows.
Dim Stop_Rx As Boolean
Private Sub btnRead_Click(ByVal sender As System.Object, ByVal e As System.EventArgs) Handles btnRead.Click
SerialPort1.Open()
[code]....
I've encounter an error which is, (Cross-thread operation not valid: Control 'txt1' accessed from a thread other than the thread it was created on.)
View 7 Replies
Jun 21, 2012
When my client try to connect to server I'm getting this error : Cross-thread operation not valid: Control 'l_users' accessed from a thread other than the thread it was created on.
Private Sub _socketManager_onConnectionAccept(ByVal SocketID As String) Handles _socketManager.onConnectionAccept
l_Users.Items.Add(SocketID)
End Sub
View 3 Replies
Feb 17, 2012
Would anyone be able to help me here please. I'm fairly new to VB.net and threading so im just trying to figure out what is happening.When I debug this I am getting the error thread operation not valid: Control 'ProgressBar1' accessed from a thread other than the thread it was created on.
I'm a little lost as to why the error is occuring or how to fix it. I've had to put the progress bar in a separate thread otherwise the GUI crashes
[Code]...
View 2 Replies
Sep 2, 2010
My understanding is that thread.join will suspend the execution of code on the calling thread until the spawned thread finishes or is aborted...
With that in mind, I tried this:
For i = 1 to 50
threads = New Thread(AddressOf test)
threads.IsBackground = True
threads.SetApartmentState(ApartmentState.STA)
[CODE]...
However, the rest of the code runs when the loop finishes, not waiting for all the spawned threads to finish. Since the rest of the code needs the threads to finish (otherwise the rest will error).
View 4 Replies
Jun 6, 2010
I have written an application that loads a form - frmCad as well as AutoCad. It does this via a class which detects when AutoCad quits.
Friend Class CadApp Private WithEvents AppObject As AcadApplication... Private Sub AppObject_BeginQuit(ByRef Cancel As Boolean) Handles AppObject.BeginQuit RaiseEvent Quit() End SubEnd Class
My main class loads frmCad as well as CadApp.
Public Class Cad Private WithEvents frmCad As CadForm Private WithEvents app As CadApp... Public Sub ShowForm() If frmCad Is Nothing Then frmCad = New CadForm frmCad.Visible = True End Sub... Private Sub app_Quit() Handles app.Quit frmCad.Dispose() frmCad = Nothing .... End Sub
When I debug the program, it stops at frmCad.Dispose()The program continues to execute, but all the code after it fails to work.Looking more carefully I get an error message which contains:-Cross-thread operation not valid: Control 'Autocad' accessed from a thread other than the thread it was created on.
View 4 Replies