Implement Multithreaded Crawler - Add A Unique Url To Each Thread To Go Process

Feb 1, 2010

I would like to implement a mulithtreaded crawler using the single thread crawler code I have now. Basically I read the urls from a text file, take each one and crawl and parse it. I know how thread basics of creating a thread and assigning a process to it but not too sure how to implement in the following way:

I need at least 3 threads and need to assign a url to each thread from a list of urls, and then each needs to go and fetch it and parse it before adding contents to a database. [Code] Now the code maynot make sense but what I need to do is add a unique url to each thread to go process.

View 4 Replies


ADVERTISEMENT

Implementing Multithreaded Crawler Using The Single Thread Crawler Code

Feb 1, 2010

I would like to implement a mulithtreaded crawler using the single thread crawler code I have now. Basically I read the urls from a text file, take each one and crawl and parse it. I know how thread basics of creating a thread and assigning a process to it but not too sure how to implement in the following way: I need at least 3 threads and need to assign a url to each thread from a list of urls, and then each needs to go and fetch it and parse it before adding contents to a database.

[Code]...

View 4 Replies

VS 2008 - Multithreaded Crawler - Each Time A New Thread Accesses One Of The Lists The Content Is Changed

Mar 18, 2010

I have written a multithreaded crawler and the process is simply creating threads and having them access a list of urls to crawl. They then access the urls and parse the html content. All this seems to work fine. Now when I need to write to tables in a database is when I experience issues. I have 2 declared arraylists that will contain the content each thread parse. The first arraylist is simply the rss feed links and the other arraylist contains the different posts. I then use a for each loop to iterate one while sequentially incrementing the other and writing to the database. My problem is that each time a new thread accesses one of the lists the content is changed and this affects the iteration. I tried using nested loops but it did not work before and this works fine using a single thread.

Here is my

SyncLock dlock
For Each rsslink As String In finallinks
postlink = finalposts.Item(i)

[CODE]...

Finallinks and finalposts are the two arraylists. I did not include the rest of the code which shows the threads working but this is the essential part where my error occurs which is basically here postlink = finalposts.Item(i) i = i + 1

ERROR: index was out of range. Must be non-negative and less than the size of the collection. Parameter name:index

I tried copying it to a new list but dosent work.

View 9 Replies

CORRECTLY Implement A Multithreaded Progressbar During A LINQ Query?

Oct 31, 2011

I have a linq query which loads approx. 1000 lines into a variable, during that process I want to display a progressbar, not necessarily stating the percentage, can be marquee style, doesnt matter.

This progressbar is on a modal form to precent the user from interacting with the app for the time the query's running.

[Code]...

View 3 Replies

Word 2007 Unique Process?

Sep 14, 2011

I'm developing an Outlook 2007 Add In with Visual Studio 2010 (VB).In the project I have a routine that opens, saves and closes Word 2007 documents.While the routine is executing, if I open Word 2007 application I see all the documents opening and closing, making it impossible for me to work in Word 2007.How can I launch the routine using a unique Word 2007 process?

View 5 Replies

Set Of Codes To Run In A New Thread Or Process?

Feb 6, 2011

how do I force a particular set of vb.net codes to run in a new thread or process.? Edit 1: I am trying TTS in vb.net but whenever click the play button , The whole program freezes and I cannot do anything else until the speech is over

View 3 Replies

Asp.net - Crawler In C# Or In .net?

Jul 24, 2009

i have a link of working crawler sample code written in either c# or in Vb.net

View 3 Replies

Event For The Termination Of A Thread And Process?

Jul 12, 2009

At the moment, I am writing a class that deals with starting an application in a thread, as well as being able to monitor the closure of the thread and process, and the creation of the thread and process through events. I have absoloutly no problems with monitering the creation of the thread and its internal process within the subs that create them. I placed notifications within the "myprocess_Exited",(for the process closure) and "myprocess_Disposed"(for the thread closure) events.

The problem comes when I terminate the outside application manually; these events do not appear to fire when the application is closed by the user, rather than by code.I thought, since the application closes with an error code when it is terminated in such a way, that it raises the "myprocess_ErrorDataRecieved" event. But this doesn't appear to happen either. Here is some of the code from that class:

'Some of the decleratoins
Event ProcessStarted(ByVal Process As Process)
Event ThreadStarted(ByVal Thread As System.Threading.Thread)

[code]....

View 3 Replies

Process Or Thread Changed Since Last Step

Mar 21, 2011

The code has gone bad. It is in debug mode and jumping around everywhere.
Error = "The process or thread has changed since last step."

Here is the code. I just create the file "start.csv" to initialize the shell command. This is very unstable for some reason. When I run line by line in debug mode it starts jumping around irrationally. It gives me versions of the above error.

Option Strict On
Option Explicit On
Imports System.IO
Module Module1
Public WithEvents Tmr As New Timers.Timer
[Code] .....

View 10 Replies

Running Process In Separate Thread

Mar 24, 2009

Here's the problem I am trying to solve. My program can export a file as a pdf. and other formats. I can open the file using process.start. My file doesnt have a good name so I am using a GUID as a name.

[Code]...

View 2 Replies

VS 2010 - How To Exit From Thread / Process

Apr 10, 2012

Basically, I am just trying to close the thread/process after it's done.

Private psi As ProcessStartInfo
Private cmd As Process
Public Delegate Sub InvokeWithString(ByVal text As String)
Private Sub Button1_Click(ByVal sender As System.Object, ByVal e As System.EventArgs) Handles Button1.Click
Dim psi As New ProcessStartInfo("m7.bat")
[Code] .....

I am trying to close the thread so once it's done, it closed for good and my form is closed as well. I tried Me.Close() which results in: "Cross-thread operation not valid: Control 'Form1' accessed from a thread other than the thread it was created on." Error.

Me.Invoke(New InvokeWithString(AddressOf CloseME), e) - Results in parameter mismatch error.
However, things like Msgbox("Done") works perfectly.

View 7 Replies

Create A Web Crawler?

Jan 18, 2010

create a web crawler using vb.net?

View 5 Replies

Creating A Simple Web Crawler?

Oct 21, 2010

I'm attempting to do a school project dealing with a simple web crawler. I have a form with a web browser control embedded that loads a web page will all the available courses. I have a text box and a button design for a user to search for a specific course by using the four letter department abbreviation. The page that I have loaded has all the department abbreviations as hyperlinks.is it possible to search the page using the four letter abbreviation specified by the user and if the search finds a corresponding hyperlink open it. Then I would use a loop to repeat the process of opening each class offered by that department and obtaining information such as course name, section number and so forth.

View 2 Replies

VS 2008 Running A External Program/process In A Thread?

Jul 7, 2011

Is it possible to run a external program/process example notepad in thread of you vb application?

View 2 Replies

VS 2010 - Process Exe Thread Is Halting Main Application?

Feb 10, 2012

I have two applications. For performing certain functions, i run a second exe, via a process, and wait till it is complete (exited) to continue with my main application. I run through many repetitive functions on different files.Everything runs fine however, when i am doing a run and it is on the x loop number, the main application seems to halt the secondary process exe. I can only assume this is what is happening as, if i close/stop (not pause) my main application in VB, the second exe continues from where it was halted until it completes.

View 4 Replies

Checking The Return Code Set By A Task / Thread / Process / Program?

Apr 16, 2010

A long time ago, on a PC far, far away from Windows, one MS-DOS program (the father) could run / execute / spawn another MS-DOS program (the son). When the son terminated, it could set a return code which could certainly be interrogated by an MS-DOS batch program or, I think, by any father program that invoked it (depending on the language in which that program was written). I have a small utilility which I want to invoke from my application, and, since that utility sets a return code, I would like to interrogate it in my application when the utility terminates. I've looked through the MSDN Library but could not find anything that seemed to cover this situation. Is there a way to do this in Visual Basic 2008 Express

View 2 Replies

Display A Modal Msgbox When Called From A Separate Thread Process?

May 14, 2010

I'm using VB.net 2005, .net framework 2.0

I'm currently having a form that has a button starting a long process, with feedback into another windows textbox (call frmProcessLog).

I'm using a Background worker to laucnh the process, upgrade the status of the process and display feedback on the frmProcessLog textbox, and to know when the thread process ended up, (from the cancel button, from an error or normal termination).

the main thread process has some msgbox in it, asking questions to the operator (yes/no/cancel).

I've added the msgboxstyle.systemmodal to the msgbox call which works "ok" excpet if the operator decide to ignore the showing message box and click the cancel button on the frmProcessLog form, which is still clickable even though I supplied "systemModal" to the msgbox. this makes the msgbox hidden behind the frmProcesslog form and the process can still continue.

If I close my frmProcessLog form, I will see the previously msgbox behind it and If I end up click one of the button it will end up at an application error because the thread that called it is not existing anymore.

So I would like that the msgbox to be launched from the thread making it as if it was launched from my frmProcessLog window in a modal way.

View 4 Replies

VS 2008 - Multi-thread Two Loops So That This Process Is Running At The Same Time?

Jun 28, 2009

It is my previous problem in vb6, its hard to multithread using vb6 so I migrate my codes to vb.Net.How can I multithread this two loops so that this process is running at the same time.Because when I start my second Loop, my first loop stop in executing. It will start again when my 2nd loop finished.

[code]...

I have 2 connected GSM Modem in usb port.That codes will send SMS in all contacts in my ListBox.

View 27 Replies

Create A Fixed-length Unique String From Another Unique Numerical Value That Is Too Long?

Feb 13, 2012

I am using VB .Net to access the eBay API and store all completed orders in a database. This database is then fed into a proprietary shipping system, which can not handle an order number larger than 20 characters. eBay returns an order number like so 230407279314-680141236013 which is too long. The order number is always 12 numbers a hyphen and 12 more numbers. What I need to do, is turn this (the result can be alpha numerical) into a shorter, unique order key to store in my database alongside the true orderId (so that this can be referenced by the shipping software instead of the actual order number). The reason for the 20 character limit is the barcode algorithm used. Is there any way to achieve this in VB .Net 2010? This number can be anything unique, so long as it does not exist already (even a good uniqueid function would work, but I would have to query the database to make sure it isn't taken)

View 2 Replies

[2008] Datagridview - Duplicate Cell Values [non-unique To Unique By Appending]

Jan 10, 2009

I'm using the datagridview. This is the sample data:

[Code]....

How do I parse through the grid to rename the duplicate values in a particular column (in this example it's the l_name field) into such a format?

[Code]....

View 1 Replies

Asp.net - Unique Number For Unique Visitor On Button Click?

Mar 21, 2011

How generate the unique no. 1,2,3 and so on .... on button click of each new user ..

the code mentioned below is a readwrite coding in vb.net ...

but the problem is it generate the same id for different users on button click event... but i want the no. of times button clicked the new ids will be generated

Protected Sub Button1_Click(ByVal sender As Object, ByVal e As System.EventArgs) Handles Button1.Click
Dim FILE_NAME As String = Server.MapPath("counts.vbi")

[Code].....

View 1 Replies

Website Crawler Creating Recursive Function To Get All The Site Link

Aug 26, 2009

i'm trying to build websites crawler and i having a bit of problem creating recursive function to get all the site link, provide a link to an example ?

View 2 Replies

Unique Constraint - Columns Currently Not Have Unique Values?

May 8, 2011

Unique constraint - these columns currently doesn't have unique values..?? I have created Unique Constraint with 3 columns. My code works perfectly but once - two users @ same time entered same data and somehow it saved in DB; after that incident. This UniqueConstraint gives me error - These columns currently doesn't have unique values. How can I check if user enters this kind of entry or how can I restrict the Entry?

View 2 Replies

VS 2010 Error - Cross-Thread Operation Not Valid: Control TbPlaca1 Accessed From A Thread Other Than The Thread It Was Created On

Aug 13, 2010

In one application, I need use 4 simultaneous threads.When the threads finished, I need to update the text of a TextBox.So, I create 4 Textbox, and I wanna the threads change the text to FINISHED.Each thread change one distinct TB.I use this "example"

Dim Terminados As Integer = 0
Private Sub frmEnviarMensagem_Load(ByVal sender As System.Object, ByVal e As System.EventArgs) Handles MyBase.Load
Timer1.Enabled = True
Timer1.Start()

[code]....

The problem is, I have an error in the red lines.The error is: INVALIDOPERATIONEXCEPTION "Cross-Thread operation not valid: Control tbPlaca1 Accessed from a thread other than the thread it was created on."I don't use cross threading. Each TB only was accessed for the correspondent thread, and for the "main program".

View 3 Replies

Cross-thread Operation Not Valid: Control 'Panel1' Accessed From A Thread Other Than The Thread It Was Created On." ?

Nov 3, 2011

This is the error message I am getting:

"Cross-thread operation not valid: Control 'Panel1' accessed from a thread other than the thread it was created on." The reason I am getting this error is because I am opening up a new form and then calling these three things:

Panel1.Show()
Label1.Show()
Label2.Show()

why I get this error message because it doesn't occur normally if I open Form2 after closing Form1, it only occurs when I open Form2 after closing Form4.

View 4 Replies

Error - Cross-thread Operation Not Valid: Control 'txt1' Accessed From A Thread Other Than The Thread It Was Created On

Jul 21, 2009

I have done a program using vb2005 to display reading from my microcontroller bs2 board but have encountered some problems. My code are as follows.

Dim Stop_Rx As Boolean
Private Sub btnRead_Click(ByVal sender As System.Object, ByVal e As System.EventArgs) Handles btnRead.Click
SerialPort1.Open()

[code]....

I've encounter an error which is, (Cross-thread operation not valid: Control 'txt1' accessed from a thread other than the thread it was created on.)

View 7 Replies

Error : Cross-thread Operation Not Valid: Control 'l_users' Accessed From A Thread Other Than The Thread It Was Created On

Jun 21, 2012

When my client try to connect to server I'm getting this error : Cross-thread operation not valid: Control 'l_users' accessed from a thread other than the thread it was created on.

Private Sub _socketManager_onConnectionAccept(ByVal SocketID As String) Handles _socketManager.onConnectionAccept
l_Users.Items.Add(SocketID)
End Sub

View 3 Replies

Threading Progress Bar In .net - Thread Operation Not Valid Control 'ProgressBar1' Accessed From A Thread Other Than The Thread It Was Created On

Feb 17, 2012

Would anyone be able to help me here please. I'm fairly new to VB.net and threading so im just trying to figure out what is happening.When I debug this I am getting the error thread operation not valid: Control 'ProgressBar1' accessed from a thread other than the thread it was created on.

I'm a little lost as to why the error is occuring or how to fix it. I've had to put the progress bar in a separate thread otherwise the GUI crashes

[Code]...

View 2 Replies

Waiting For Threads - Thread.join Suspend Execution Of Code On Calling Thread Until Spawned Thread Finishes Or Is Aborted

Sep 2, 2010

My understanding is that thread.join will suspend the execution of code on the calling thread until the spawned thread finishes or is aborted...

With that in mind, I tried this:

For i = 1 to 50
threads = New Thread(AddressOf test)
threads.IsBackground = True
threads.SetApartmentState(ApartmentState.STA)

[CODE]...

However, the rest of the code runs when the loop finishes, not waiting for all the spawned threads to finish. Since the rest of the code needs the threads to finish (otherwise the rest will error).

View 4 Replies

Cross-thread Operation Not Valid: Control 'Autocad' Accessed From A Thread Other Than The Thread It Was Created On

Jun 6, 2010

I have written an application that loads a form - frmCad as well as AutoCad. It does this via a class which detects when AutoCad quits.

Friend Class CadApp Private WithEvents AppObject As AcadApplication... Private Sub AppObject_BeginQuit(ByRef Cancel As Boolean) Handles AppObject.BeginQuit RaiseEvent Quit() End SubEnd Class

My main class loads frmCad as well as CadApp.

Public Class Cad Private WithEvents frmCad As CadForm Private WithEvents app As CadApp... Public Sub ShowForm() If frmCad Is Nothing Then frmCad = New CadForm frmCad.Visible = True End Sub... Private Sub app_Quit() Handles app.Quit frmCad.Dispose() frmCad = Nothing .... End Sub

When I debug the program, it stops at frmCad.Dispose()The program continues to execute, but all the code after it fails to work.Looking more carefully I get an error message which contains:-Cross-thread operation not valid: Control 'Autocad' accessed from a thread other than the thread it was created on.

View 4 Replies







Copyrights 2005-15 www.BigResource.com, All rights reserved