Im making a vb.net app and im using htmlagilitypack. I need hap to get the profile links from [URL]
Here is an example of the html:
<a href="/bus/Ontario/Brampton/A-Safe-Self-Storage/17142.html?what=af&where=Ontario&le=1238793c7aa%7Ccf8042ceaa%7C2ae32e5a2a" onmousedown="utag.link({link_name:'busname', link_attr1:'in_listing_left',
[Code].....
There are placenames on this webpage and if you click on the placename it leads you to its profile. I need to get all the profile links into a textbox on my form. An example of the webpage [url]...
View 3 RepliesI want to download a webpage by httpWebRequest.
Web page Address >> http://www.stockbangladesh.com/resou...l=18455&inv=60
from a free website
[URL]
But that page requires login first. my user name : babul37 and pass : mhbb7337 How can I download that page by httpWebRequest?
I have browsed through many posts on this and have tried some of the suggestions but still not understanding it fully. I would like to scrape html pages that have some script running that usually executes the script to display a link after clicking. Some mentioned firebug and others talked about reverse engineering the code I need. But after trying reverse engineering I still dont see how to get the data after tracing the script function.[code]I am using vb.net and some sites were easy using firebug where looking at the script I was able to pull the data that I needed. What woudl I do in this scenario? the link is url... and the categories are what I am trying to access. Notice the url does not change.
View 1 Repliesworking on some code that will screen scrape a site then fine some code in there and pull it out for me...but for some reason i cant find the right regular expression. Anyone can with creating a regular expression that will match
/clipmark/C51BC80B-173F-4BC7-B630-CCDB8B56C9DF/
Have tried multiple attempts of writing it and each time it doesnt bring back all of the content or the wrong content.
I got a small issue. Im trying to grab some links(about 5 only) from a webpage that can change frequently.
Im using:
For Each ClientControl As HtmlElement In wb.Document.Links
ListBox1.Items.Add(ClientControl.GetAttribute("href"))
Next
It gets the value of the link the files are (the hyperlink) and allows me to download the file, but I want to get to get the string assocaited with it as well
For example, A link says click here! and bring you to a page.
I can get the link to the page, but not the text click here according to my source code.
I'm actually trying to code a downloader for a site that generate download links.The program can download one link, but when there are more than one link, it only downloads the first one.
[Code]...
The thing that I am trying to achieve is -
1) Create a simple Web Browser [DONE]
2) Clear Cookies by Mouse Click on the Button [DONE]
3) Save the links present in the webpage that the user is viewing through my browser [PROBLEM]
Here's the code that I have done -
Code:
Imports System.IO
Private Sub Button1_Click(ByVal sender As System.Object, ByVal e As System.EventArgs) Handles Button1.Click
[Code]......
I'm trying my hand at making a program that crawls some of the larger retail sites for prices of products in a single category, TV for example. With Amazon, Best Buy, Walmart, Newegg, none of them are an issue so far because I can call the site using httpwebrequest and parse with regex.The one that has me completely stumped is Target. If you check this link: url...You can see that there is a grid of all the TVs they sell. The problem is that the links are not part of the original page as they do not appear when you view the source code. As far as I can tell, they're called in by some Javascript code when you load the page.From what I've gathered, loading this final content via httpwebrequest is impossible. So how are pages like this crawled? Would a workaround involve using built in webbrower tool in vb.net? That seems clunky to me. Anyone point me in the right direction
View 1 RepliesI'm just curious as to how some software programs that I see out there have the ability to extract links & text from thousands of web pages at an extremely high and fast rate. Has anyone here, ever created a link or text extracting program the has the ability to parse many webpages and return data into a textbox? I know how to extract links via the webbrowser control, but it doesn't seem to parse/extract data at a very high & fast rate like many email, link & text extracting programs that I see out there.
[Code]...
I am trying to make a project, but I am beginning and the examples that I see in the forums are giving certain. desire to make a program where web opens definitive page. When opened, the shown content is copied and recorded in an archive. Desire not to copy code HTML, but yes, the text that is appearing in the screen.
View 6 Repliesmy webbrowser navigates to a webpage. I need to store all the links into a collection, I did find the code to do the job (on this forum) and it works, but there is a problem: there are more links on the page than those the code reads, like when i right click on a picture and choose "copy shortcut" I get a link that is not showing when I display the "view source" for the entire page. I can't figure out how to do it.
View 3 Repliesi am trying to parse an excel file. its structure is very complex. The possible way i know are.
Use Office introp libraries
Use OLEDB provider and read excel file in a dataset.
But the issue is of its complexity like some columns,cells or rows blank etc. What are the best possible ways to do this?
i need to scrape a p tag which has h3 tag after it but does not have a closing p tag. It looks like this :
<script ad>asdasdasd</script>
<p>Translation companies are
-----------------------
-----------------------
<h3 class="this_class">mind blown site</h3>
There is no </p> tag so i cannot parse it completely. Now i have two questions :
1) can this be parsed using httpagility xpath ?
2) i have a function to find text between two strings (getbetween). But i have a doubt - If i use "asdasdasd" and " is it always 100% that vb.net will use the script tag which is just above h3 because there are 2-3 same lines - "asdasdasd"
I have a windows forms application that scrapes a website to retrieve some data. I would like to implement the same functionality as a windows service. The reason for this is to allow the program to run 24/7 without having a user signed in.
To that end, my current version of the program uses a web browser control (system.windows.forms.webbrowser) to navigate the pages, click the buttons, allow scripts to do their thing, etc. I cannot figure out a way to do the same without the web browser control, but the web browser control cannot be instantiated in a windows service (because there is no user interface in a web service).
I was amazed with how little documentation or examples are out there for checking the pagerank for a website using vb.net . I was hoping that someone might have some example code or other resources regarding how to do this (using toolbarqueries.google.com - not by scraping data from a 3rd party website).
View 3 RepliesI have been struggling very much trying to figure out how to get my vb program to enter a comment into myspace's comment box...I am trying to get this working cause I am managing some myspace pages for some bands/artists and would like to automate the commenting for new friends.When I automate the filling of a webpage's textbox, I use this kind of
webbrowser1.document.getelementbyid("HTML ID/NAME").setattribute(value, mycomment.text)But this seems to not work with the html elements on myspace's comment textbox...The html code I found for myspace's comment box is this:
<div contenteditable="true" class="smartEdit" data-tip="Post a comment...">Post a comment...</div>
what code to use to get the vb program to fill out the comment box on myspace,
I trying to get text from webbrowser to textbox but it wont work..
<strong><span id="SPANID">VALUE</span></strong>
I trying to get VALUE in textbox.
i have two pages the first to insert personal data and the other page cridet data when i enter ther personal data and go to the other page the credit page i want to take the data in textbox control in the first page how?? i tried the way but it doesn't work
i use visual basic dim a as string a = personal.textbox.text
Im trying to get school information of a site and want to save it as an excel tables sheet with details in each column, inital start was the below code help me in going further.Column headers: school name, mascot, address, type, phone, fax etc. for a list of schools I have. For example I used one link.
[Code]...
Is there anyway to access internet sites if i code in VB6. i have gone thru a few basic tutorials and didn't found any command/method to develop or code a program which can access internet and do my work.
View 6 RepliesI am making scraper that uses url list. But I have no idea how to pass that wanted url from that list to thread that does scraping.
Public Sub init2_check()
Dim t(source1.Items.Count()) As Thread
For x = 0 To UBound(t) - 1
t(x) = New Thread(AddressOf ScraperDoWork)
t(x).IsBackground = True
t(x).Start(x)
thread.Sleep(100)
Next
Me.Invoke(New button_Delegate(AddressOf enable2_button), btn_StartScrape)
End Sub
I have just made a program were i enter a proxy list then my software views a url over and over using the proxys . I was using a web browser in side the program but worked out slow.So i now have changed the code to htpp request. But now when i run the program it goes up by 2 the views then frezzes .Here is my code
rivate Sub Timer1_Tick(ByVal sender As System.Object, ByVal e As System.EventArgs) Handles Timer1.Tick
If ListBox1.Items.Count = 1 Then
[code].....
i want to link text box of my software with the text box of the web page (i.e values entered in my software should be linked to values of web page E.g. username entered in software should be linked with username entered in yahoo/facebook site)
View 1 RepliesIm working on an app that will scrape addresses from yellowpages.ca
View 5 Replieshow to SetAttribute (a Value) for a textbox in a webpage document that don't have an ID .I already knows that to SetAttribute to a textbox you just need the ID of the element.
I.E : WebBrowser1.document.GetElementByID(THE ELEMENT ID).SetAttribute("Value",TextBox1.text)
So as you see in the example, to set attribute to an Element I need an ID and in my case I don't know what is the ID of the Specific Element.
This is the ViewSource of the Textbox :
<input name="ctl00$ContentPlaceHolder$ccEnterCode" type="text" size="5" maxlength="5" value="">
I don't see where's the ID !
I have a Web Browser and I have a button where I can get the HTML Source of a Web Page. The code I have so far stores the HTML Source in a string:
[Code]...
What I need to do is grab all the web page elements on a web page (id, name, etc) and place them into a textbox to view...Currently I have this
Dim htmlElements As HtmlElementCollection = WebBrowser1.Document.GetElementsByTagName("input")
For Each el As HtmlElement In htmlElements
If el.GetAttribute("name").Equals("name") Then
[code]....
I know how to browse page witht he browser control etc, but i need to know how to grab data from within the html.
[Code]....
i have a textbox in my app, and when you click the button it inputs the textbox.text into a website textarea this worked fine when i was testing it using the source code (html file on my computer) but what i did not realise was that the textarea was formatted with some java (added tools to the texarea) so now when i try it on the website it does not work as the textarea has the java running. (calls it from another url) code i was using
Me.WebBrowser1.Document.GetElementById("elm9").SetAttribute("value", Form1.TextBox10.Text)
IN THE HTML:
textarea
<td>
<textarea name="home" id="elm9" style="height:400; width:900">