Implementing Multithreaded Crawler Using The Single Thread Crawler Code
Feb 1, 2010
I would like to implement a mulithtreaded crawler using the single thread crawler code I have now. Basically I read the urls from a text file, take each one and crawl and parse it. I know how thread basics of creating a thread and assigning a process to it but not too sure how to implement in the following way: I need at least 3 threads and need to assign a url to each thread from a list of urls, and then each needs to go and fetch it and parse it before adding contents to a database.
[Code]...
View 4 Replies
ADVERTISEMENT
Feb 1, 2010
I would like to implement a mulithtreaded crawler using the single thread crawler code I have now. Basically I read the urls from a text file, take each one and crawl and parse it. I know how thread basics of creating a thread and assigning a process to it but not too sure how to implement in the following way:
I need at least 3 threads and need to assign a url to each thread from a list of urls, and then each needs to go and fetch it and parse it before adding contents to a database. [Code] Now the code maynot make sense but what I need to do is add a unique url to each thread to go process.
View 4 Replies
Mar 18, 2010
I have written a multithreaded crawler and the process is simply creating threads and having them access a list of urls to crawl. They then access the urls and parse the html content. All this seems to work fine. Now when I need to write to tables in a database is when I experience issues. I have 2 declared arraylists that will contain the content each thread parse. The first arraylist is simply the rss feed links and the other arraylist contains the different posts. I then use a for each loop to iterate one while sequentially incrementing the other and writing to the database. My problem is that each time a new thread accesses one of the lists the content is changed and this affects the iteration. I tried using nested loops but it did not work before and this works fine using a single thread.
Here is my
SyncLock dlock
For Each rsslink As String In finallinks
postlink = finalposts.Item(i)
[CODE]...
Finallinks and finalposts are the two arraylists. I did not include the rest of the code which shows the threads working but this is the essential part where my error occurs which is basically here postlink = finalposts.Item(i) i = i + 1
ERROR: index was out of range. Must be non-negative and less than the size of the collection. Parameter name:index
I tried copying it to a new list but dosent work.
View 9 Replies
Jul 24, 2009
i have a link of working crawler sample code written in either c# or in Vb.net
View 3 Replies
Jan 18, 2010
create a web crawler using vb.net?
View 5 Replies
Oct 21, 2010
I'm attempting to do a school project dealing with a simple web crawler. I have a form with a web browser control embedded that loads a web page will all the available courses. I have a text box and a button design for a user to search for a specific course by using the four letter department abbreviation. The page that I have loaded has all the department abbreviations as hyperlinks.is it possible to search the page using the four letter abbreviation specified by the user and if the search finds a corresponding hyperlink open it. Then I would use a loop to repeat the process of opening each class offered by that department and obtaining information such as course name, section number and so forth.
View 2 Replies
Aug 26, 2009
i'm trying to build websites crawler and i having a bit of problem creating recursive function to get all the site link, provide a link to an example ?
View 2 Replies
Aug 10, 2010
Is it possible to automatically insert a Code Snippet when an interface is implemented? If so, how do you do it? I am looking for something similar to when you implement IDispoable in VB.[code]This will be used by web forms when transfering parameters from one page to the next using Server.Transfer
View 1 Replies
Dec 23, 2009
Say I have a code with 3 methods that do some pretty intensive work. Would executing these methods on 3 seperate threads be faster then executing them one after the other on a single core cpu? And what if it's a dual core or HT?
View 8 Replies
Nov 14, 2009
I am getting this exception: Cross-thread operation not valid: Control 'TextBoxOutput' accessed from a thread other than the thread it was created on. in a VB 2008 Windows Forms application that I havent specifically made multi-threaded. I have create a sub called Out() like this:
Private Sub Out(ByVal inString As String)
Me.TextBoxOutput.Text += (Me.TextBoxOutput.Text & inString & vbNewLine)
Me.TextBoxOutput.SelectionStart = Me.TextBoxOutput.Text.Length
Me.TextBoxOutput.ScrollToCaret()
End Sub
and I call it from various events to log messages I am getting from my pbx, which I use a third party open source dll to access. The interesting thing to me is that if I run the deployed application (ie, not in the VS IDE) I do not get any exceptions, it's only from within the IDE.if I add the line: Control.CheckForIllegalCrossThreadCalls = False to the form's constructor, the exceptions stop, but a) this seems like a hacky way to get around the issue and b) the application seems to hang sometimes when I have it set this way.
View 2 Replies
Apr 21, 2010
I'm getting the following error:"Current thread must be set to single thread apartment sta mode before ole calls can be made"
I am using Speech.Recognition.SpeechRecognizer and using it to open various dialog boxes. It works fine for the first few but on the third one when it calls the InitializeComponent in the Dialog's New, it throws the above error.
Anybody have any clue what causes this or how to fix it? The SpeechRecognizer seems to run on its own thread, so that seems to be at the root of the problem. If I open thedialog with a button click I don't get an error.
[Code]...
View 1 Replies
Sep 2, 2010
My understanding is that thread.join will suspend the execution of code on the calling thread until the spawned thread finishes or is aborted...
With that in mind, I tried this:
For i = 1 to 50
threads = New Thread(AddressOf test)
threads.IsBackground = True
threads.SetApartmentState(ApartmentState.STA)
[CODE]...
However, the rest of the code runs when the loop finishes, not waiting for all the spawned threads to finish. Since the rest of the code needs the threads to finish (otherwise the rest will error).
View 4 Replies
Feb 26, 2010
I have an interface that I've defined and am working through a bunch classes that will implement it. I'm noticing that in 95% of the implementations, I'm only changing a few lines of the code, so I'd like to know if it's possible to have my IDE generate a block of code inside of a method upon implementation, much the same as implementing IDisposable.not mark my posts as answered. It is extremely rude. You have no idea if you've answered my question.I WILL come back and mark the ones that are answers, as answers.
View 7 Replies
May 8, 2009
I am having trouble with this project I am working on. I am a beginner to vb and have some background in access databases. I will attach my am doing an inventory project. When you execute the form, a main form pops up that allows you to choose Production or Inventory. My issue is with the Production. In the production form that pops up after clicking Production, there are text boxes to be filled out by the user based on the production run.plementing a class that will give the box count based on the box type entered. I have to use a class based on the project requirements.
Basically to calculate the usage, enter in a random number for Pods Produced, then enter in a box type (which is A box, B box, C box, D box, Master Case). I want the Public Class Box Count in the code to be able to be placed in the production form's main class code and calculate the box count based on the box type entered. If A box, box count should be 50. If B Box, box count should be 100, If any other box is typed in, the box count should be 150
View 4 Replies
Dec 17, 2009
i know that a single function could not be run until it has finished and is currently not running. what if i have 2 functions within a class, are both of them allowed to run simultaneously? or will running 1 of them prevent both of them from running until it is done?(assume that backgroundworkers and threads are running the functions of this class)
EDIT: plus what if a single function has multiple signatures, is it considered a single function or 2 different functions? could both of them be running at the same time or only 1 is allowed to run between the 2 at any one time?
View 13 Replies
Oct 20, 2009
As my latest questions have stated, i am in a process of researching on Silverlight 3 and its application as a suitable RIA solution to a pre-built project a client of ours wishes to webify. And my experience in .Net and silverlight is about 3 - 4 weeks.I have now become aware of .Net RIA Services. I require to use an existing VB class; which was created as a quick wrapper class to emulate a small piece of the client's application code. This is used as a proof of concept. It contains one function which uses the client's DLL files to return a simple data type This class has already been used successfully with Flex/Weborb combination.I unfortunately do not have enough knowledge or experience in Silverlight/RIA Services to grasp what it is i have to do to get the file working. All the examples I find online only specify the method where you use a database as dataSource etc.Could you please inform me how i would go about implementing RIA Services with Silverlight 3, using existing code/classes with no DB? Or could you point me in a direction for further research on the matter?
View 1 Replies
Jun 5, 2011
I'm having a small issue; I start a process and when that process exits, I raise an event which removes a control from my form.It all works perfectly until the part where I want to remove a control from my form. Visual Studio gives me a cross-thread exception even though I'm programming single threaded.
View 6 Replies
Oct 15, 2010
With the DataGridView I'm used to just populate them with a list of some object, using the DataSource property. Now I want to just display properties from a single object, and I'm a bit confused on how to do it.. The object is updated through an API event, and I want to select some of the properties for displaying in the DataGridView, to look like this:.where "ValueX" is the name of the properties.Now, what would be my options for binding this to the DataGridView, such that property changes are automatically updated. Since an update also must invoke the control, does that mean I must reassign all bindings, or what logic is necessary in this case?
should I bind to an array, arrayList, stringList, dataTable etc? I guess I'm not sure which type of source will allow binding. Or if I should perhaps build the columns and cells of the grid and populate them "one-by-one" to the properties of the object?
View 6 Replies
Jul 19, 2011
This code was posted in Chit Chat and everyone is saying how great it is. I just have Visual Studio 2010 and no familiarity with VB6 so I thought it would be good to convert the code to Visual Basic 2010.
[Code]...
View 39 Replies
Jul 24, 2009
I am trying to create a new thread that will import from an excel spreadsheet, because it was doing all the import before the form actually displayed to the screen. So the user never saw the form while it was importing..In the form's load event I create the thread and start it, but I get this large error:An error occurred creating the form. See Exception.InnerException for details. The error is: ActiveX control '8856f961-340a-11d0-a96b-00c04fd705a2' cannot be instantiated because the current thread is not in a single-threaded apartment.
I am using visual studio 2005 pro on Vista Business?
View 2 Replies
Aug 13, 2010
In one application, I need use 4 simultaneous threads.When the threads finished, I need to update the text of a TextBox.So, I create 4 Textbox, and I wanna the threads change the text to FINISHED.Each thread change one distinct TB.I use this "example"
Dim Terminados As Integer = 0
Private Sub frmEnviarMensagem_Load(ByVal sender As System.Object, ByVal e As System.EventArgs) Handles MyBase.Load
Timer1.Enabled = True
Timer1.Start()
[code]....
The problem is, I have an error in the red lines.The error is: INVALIDOPERATIONEXCEPTION "Cross-Thread operation not valid: Control tbPlaca1 Accessed from a thread other than the thread it was created on."I don't use cross threading. Each TB only was accessed for the correspondent thread, and for the "main program".
View 3 Replies
Nov 3, 2011
This is the error message I am getting:
"Cross-thread operation not valid: Control 'Panel1' accessed from a thread other than the thread it was created on." The reason I am getting this error is because I am opening up a new form and then calling these three things:
Panel1.Show()
Label1.Show()
Label2.Show()
why I get this error message because it doesn't occur normally if I open Form2 after closing Form1, it only occurs when I open Form2 after closing Form4.
View 4 Replies
Jul 21, 2009
I have done a program using vb2005 to display reading from my microcontroller bs2 board but have encountered some problems. My code are as follows.
Dim Stop_Rx As Boolean
Private Sub btnRead_Click(ByVal sender As System.Object, ByVal e As System.EventArgs) Handles btnRead.Click
SerialPort1.Open()
[code]....
I've encounter an error which is, (Cross-thread operation not valid: Control 'txt1' accessed from a thread other than the thread it was created on.)
View 7 Replies
Jun 21, 2012
When my client try to connect to server I'm getting this error : Cross-thread operation not valid: Control 'l_users' accessed from a thread other than the thread it was created on.
Private Sub _socketManager_onConnectionAccept(ByVal SocketID As String) Handles _socketManager.onConnectionAccept
l_Users.Items.Add(SocketID)
End Sub
View 3 Replies
Feb 17, 2012
Would anyone be able to help me here please. I'm fairly new to VB.net and threading so im just trying to figure out what is happening.When I debug this I am getting the error thread operation not valid: Control 'ProgressBar1' accessed from a thread other than the thread it was created on.
I'm a little lost as to why the error is occuring or how to fix it. I've had to put the progress bar in a separate thread otherwise the GUI crashes
[Code]...
View 2 Replies
Jun 6, 2010
I have written an application that loads a form - frmCad as well as AutoCad. It does this via a class which detects when AutoCad quits.
Friend Class CadApp Private WithEvents AppObject As AcadApplication... Private Sub AppObject_BeginQuit(ByRef Cancel As Boolean) Handles AppObject.BeginQuit RaiseEvent Quit() End SubEnd Class
My main class loads frmCad as well as CadApp.
Public Class Cad Private WithEvents frmCad As CadForm Private WithEvents app As CadApp... Public Sub ShowForm() If frmCad Is Nothing Then frmCad = New CadForm frmCad.Visible = True End Sub... Private Sub app_Quit() Handles app.Quit frmCad.Dispose() frmCad = Nothing .... End Sub
When I debug the program, it stops at frmCad.Dispose()The program continues to execute, but all the code after it fails to work.Looking more carefully I get an error message which contains:-Cross-thread operation not valid: Control 'Autocad' accessed from a thread other than the thread it was created on.
View 4 Replies
Feb 9, 2011
My error:Cross-thread operation not valid: Control 'label2' accessed from a thread other than the thread it was created on. I want to be able to give my Public RunBot() sub access to edit Labels, buttons, etc without an error.
[Code]...
View 8 Replies
Apr 1, 2010
I'm getting the 'System.InvalidOperationException' error on acccessing a thread from other than where it was created. My code is:
'Required by the Windows Form Designer
Public components As System.ComponentModel.IContainer ' changed from private
[code]....
View 3 Replies
Jun 6, 2009
My problem is Cross-thread operation not valid: Control 'statusHO' accessed from a thread other than the thread it was created on.
System.InvalidOperationException was unhandled
Message="Cross-thread operation not valid: Control 'statusHO' accessed from a thread other than the thread it was created on."
Source="System.Windows.Forms"
StackTrace:
[Code]...
i used it and it works fine then. but as it is mentioned its not a good practice to use it. So what should i do? is it ok if i use this as i am not able to use correct practice. Mine application is not for enterprise purpose. its just my class project, and it works fine even if i put application working in background. System works ok. I dont know how to correct this error.As i dont have that much knowlege of threads.
So please guide me how to fix this in correct way otherwise i am force to go for bad programming practice
Please refer to Windows Mobile Developer Center > Smart Device Development Forums > Windows Mobile Development > Appendtext not working for all my code details
View 1 Replies
Aug 4, 2009
Cross-thread operation not valid: Control 'lv1' accessed from a thread other than the thread it was created on.
Dim lvi As ListViewItem
For Each lvi In lv1.Items
[code]....
View 1 Replies
