I'm writing a web crawler for a specific site. The application is a VB.Net Windows Forms application that is not using multiple threads - each web request is consecutive. However, after ten successful page retrievals every successive request times out. I have reviewed the similar questions already posted here on SO, and have implemented the recommended techniques into my GetPage routine, shown below:
[Code]...
1) How do I use cookies obtained from login to do subsequent requests? e.g. Login to a forum, then use the cookie obtained from login to access User CP and change settings.
2) How do I parse HTML via HttpWebRequest? Does mshtml work with HttpWebRequest? If so, how do I work with it? I thought of downloading the source code of the page I'm requesting into a richtextbox and do my stuff from there, but it sounds kinda impractical to me since I have to use regex to get the innertext of stuff (or not?).
What I would like to know is how to obtain form input elements during an httpwebrequest?
[Code]...
Is it possible to obtain these input elements during an httpwebrequest without having to view the html code?
I'm using HTMLAgilityPack in a parser that I have up on a server, but I'm having issues with one of the websites that I'm parsing: Every day around 6am they tend to shut down their servers for maintenance, which throws off the Load() method for HTMLWeb, and makes my app crash. Do any of you guys have a more secure way of loading a website into HTMLAgilityPack, or maybe some way to do error checking in C# to prevent my app from crashing? (my c# is a little rusty). Here is my code right now:
HtmlWeb webGet = new HtmlWeb();
HtmlDocument document = webGet.Load(dealsiteLink); //The Load() method here stalls the program because it takes 1 or 2 minutes before it realizes the website is down
The current version of my application was using the WebClient object to download files from the internet and all was working well until some users reported that they were getting server timeouts. I looked into this issue and decided to move to the WebRequest object as it allows me to specify a Timeout value. The problem is, when using it synchronously, if I specify a timeout value of -1, I am worried that in the event of some weird server anamoly, my application would hang as it would be waiting for a response that isn't coming. So then I thought of using the WebRequest ansynchronously.
[Code]....
In an asp.net 2.0 application I maintain, we were experiencing issues with transaction aborted errors (timeout). Code that fails seems to cause a timeout, then the page logging functionality that uses transactionscope (default constructor) fails (but not always). The timeout is set to 2 minutes. Some example code that is similar to what we have in our app is below:[code].....
From what I've seen and read, my guess is that since the using block is never reached, the transaction times out.Then the logging code (done with any page request) tries to enlist in the existing transaction, which has timed out and results in a transaction aborted error (as soon as the constructor is called). Is this assumption correct? Why would only some requests fail going forward and not all (assuming they all use transactionscope)? how does connection pooling play into this? If user A hits the bad code, would user B then be potentially affected by this? This is the behavior we've seen. If not, what else might cause this? I've been to MSDN, but I can't find anything that really clicks with me as to what is happening and why.Here is the relevant portion of the connection string:
Enlist=true;Pooling=true;Connection Lifetime=20;Max Pool Size=25;Min Pool Size=5
FYI. Not sure if this is relevant, but the app uses an Oracle 11g database with the EntLib database factory pattern.
I'm running some unit tests for a large VB.Net application I help maintain and am running into problems with tests timing out. The test run is set up so that there's no timeout limit on the overall test run, but a limit of 2 minutes per individual test. However, a couple of the tests take a lot longer than this to run, so we tried adding the timeout attribute to the test's declaration, i.e
<TestMethod(), Timeout(15 * 60 * 1000)> Public Sub DoSomething()
for a 15 minute timeout. However, the 2 minute timeout set in the test run properties seems to be taking precedence, even though it says that the timeout attribute should override it.
I have been using the following Stream Reader code for over a year now to pull information from 3 websites w/ no issues. Recently my users are getting massive amounts of "Timeout" errors. I have opened a ticket w/ the IT department but they have not found any issues (we are experiencing off/on issues w/ our web traffic speeds in our department). Per the code I have the timeout set to 20 seconds but the Timeout error occurs in only a few seconds from the System.Net.WebException.
Imports System.Text.RegularExpressions
Imports System.Net
Imports System.IO
[code]....
i want to check whether the dates in a file are consecutive.that is if the dates are 7.10,2011 ,7.11.2011,7.12.2011 it should return true and if they are 7.10.2011,7.12.2011 it should return false.
View 8 RepliesI have code in an ASP.NET form that needs to, depending on user entry create messages in the database. We are speaking of potentially thousands of db entries. How do I protect against deadlocks, I mean apart from using Transactions and setting IsolationLevel to Serializable, as well as using WITH(NOLOCK) statement on my select statements since I don't mind a dirty read.
[Code]...
I need to find consecutive numbers in an array and return a string which tells the range and numbers that don't form a range.I found some of the already asked questions but none of them is in VB.Net:Add to array consecutive numbersIf the array of numbers looks like {11,12,67,68,69,70,92,97} then returned string should be of the form 11,12, 67 through 70, 92 and 97.
View 1 RepliesThe problem is to find if a word has 3 consecutive letters in it. Such as THIRSTY, STUDENT. Here's the code I've written, but I'm stuck on how to actually write the code to check the letters.
Private Sub btnRun_Click(ByVal sender As System.Object, ByVal e As System.EventArgs) Handles btnRun.Click
Dim word As String = txtWord.Text
[Code].....
This is my first post on here--I've been teaching myself VB via forums like this and MSDN for about 2 years now. I'm programming a computerized economics experiment, and I'm using a communications package that a guy I worked with developed. He and I have been going back and forth trying to figure out what the heck is going on.
The upshot of the communications package is that there's a serializeable MQMessage class that we send back and forth, essentially just a package of an integer (Type), string (Text), and Object (Data). Most of the time, this goes according to plan.
The problem is this: If the client receives two or more messages back-to-back, the client stops raising the MessageArrived event. (and presumably, the same would hold true for the server--I should test this, but jeez-oh-man, I've been running test after test for the last week).It's still connected, as on disconnect it throws the "Socket forcibly closed" exception. When it receives multiple messages sequentially, it's still receiving the data, it just never knows when the first message ends, and the buffer just keeps filling and filling. The problem seems to be that it can never successfully Deserialize the MQMessage object, and so it returns Nothing on the getCompletedMessage function and just keeps adding to the buffer (ABuffer.Length goes 1024, 2048, etc.).
I'm trying to load some data from a MySQL table and sort it so two equal rows doesn't come consecutive. I've been trying to do this on the server-side, but it looks like it's impossible. So I'm wondering if it's possible to load the data into a dataset and sort it there or something like that? I'm open for all solutions, both on the server-side and client-side.
[Code]...
This was a difficult question to google. I have an Order class with a couple of dozen properties such as ClientReference, ShipMethod, etc. I use databinding to bind this class to an Order form. If a user enters an invalid shipping method, an exception is raised in the ShipMethod property and a validation error is displayed to the user. But I'm trying to use the same Order class within an OrderImport class whose job it is to take a flat file and create multiple orders. The problem occurs if I'm assigning to a dozen of the order fields consecutively, and say the first one causes a validation exception. If I have all dozen assignments occuring within a single try-catch block, the next 11 assignments will be skipped and I'll only be able to record one validation error. I could solve the problem by doing a dozen separate try-catch statements, but that doesn't seem very elegant.
View 22 RepliesI am using VB.NET's TextFieldParser (Microsoft.VisualBasic.FileIO.TextFieldParser) to read a delimited file. However, when I try to read in a field with consecutive newlines within the field, the consecutive newlines are turned into a single new line. I would like the consecutive newlines to be preserved, but am unsure how.Here's an example file that I am reading in with exactly one field. The quotes are part of the file's content and there are three newlines (including the two consecutive newlines following line 2):
"This is line 1
This is line 2
This is line 4, which follows two consecutive newlines."Here is the code that I am using to parse and read in the file:
Dim reader as New Microsoft.VisualBasic.FileIO.TextFieldParser(myFile, System.Text.Encoding.Default)
reader.TextFieldType = FileIO.FieldType.Delimited
reader.SetDelimiters(",")
[code].....
I'm trying to make a multi-threading app which extracts the names from my site. I have 1000 pages celebrity related. The urls are on a listbox List1 and the number of threads are specified in TextBoxThreads I would like each thread take care of a coresponding url on list, when they finish go to the next one and so on (for each thread a url)
[Code]....
I have a vb.net web application and when a particular function runs , i get data timeouts in the rest of the application..(ie..row not found errors or column does not belong to table but it does) The function is adding multiple rows in multiple tables in the database and is running in a for loop. It seems to be all SQL related but I am not seeing anything in the error logs in SQL or in the application Right now I am assuming it is memory related where to start note..the for loop will be replaced with a bulk insert but right now I jest need to resolve the issue of the timeouts
View 1 Repliesexplain to me how listening to ARP requests in VB.net or C #?
View 2 RepliesI have a website and say a user comes along and fills out the form with bogus data. Then after he fills it out, he enters another one and another one and another one of bogus data. He's basically just trying to clog up the data base.
How is the best way to combat this? Would it make sense to when they submit the data track the IP address?
I'm trying to make a program that will calculate the sum of consecutive numbers between two given numbers. [code]
View 13 Repliesi am creating a httpRequest handler to download files my question is which better to use aspx or ashx ?
View 3 RepliesI'm wanting to write a program in vb .net that could monitor outgoing print requests from any program on the PC, prompt the user if they would like to add a header to the document, and if they say yes, then tack it on (if the user has printed something in the space where the header will go, just print on top of it). Is this possible? I've looked around for ideas on how to do this, but I can't seem to find anything.
View 1 Repliesthis is general to any operation calling an SQL server, or anything requiring an open connection at that.
Say I have anywhere from 20 to 1000 Select calls to make for each item in data being looped. For each step, I'll select from sql, store data locally in a struct, then proceed. This is not a very expensive call, so should I keep the connection open for the entire loop? Or should I open and close every step?
How expensive in run time is opening a connection? I would think it'd be better to keep the connection open, but would like to get the correct response for this.
while using HTTP web Requests, when i try and read the stream i always get an error saying 403 Forbidden but if i try to do it in the VB.Net web browser it works fine. Here is my code:
Imports System.Net
Imports System.Text
Imports System.IO
[Code]....
I have a server making a head request to a database dump I've created. The remote server does this to make sure that it's not using excessive bandwidth when not necessary.However, due to some other circumstances outside my control this causes the script to be hit twice: once for the head request, and then another time to download the data.
What I'd like is to have the script I've written detect the head request, send back a couple of headers (e.g. last modified is right now, filesize different than before), and exit. Is there a way to do this?
I have a server making a head request to a database dump I've created. The remote server does this to make sure that it's not using excessive bandwidth when not necessary.However, due to some other circumstances outside my control this causes the script to be hit twice: once for the head request, and then another time to download the data.
View 1 RepliesI have a web service I'm trying to interface with however no matter what I try it is refusing to send the clients credentials along with the request.The security block in my app.config looks like
<security mode="Transport">
<transport clientCredentialType="None" proxyCredentialType="None" realm="" />
<message clientCredentialType="UserName" algorithmSuite="Default" />
[code].....
I have a site developed using vb.net 2.0. It has a URL of like [URL]..I want to avoid any malicious requests using other URLs. I have tried to implement using the code below and would like to kick such requests out of the site on the page load. How can I achieve this?
If (urlParams.Contains("http") And
Not urlParams.Contains(Request.ServerVariables("HTTP_HOST").ToString)) Then
Would like to kick out of the site here.
End If
In VB.Net, how do I make a program to tap into the HTTP requests made by external web browsers? I don't want to edit or stop the headers, only 'read' them, i.e., access its contents. I'm speaking of something like Fiddler2, but I only want the 'reading' part. I also want to read the user-agent.
View 1 Replies