Recursive AI

Extract all links and images from a page

All onehtml recipes

List every link and image on a page with its resolved URL, its attributes in source order, and the accessible name a screen reader announces for it.

C#
using RecursiveAI.OneHtml;
using RecursiveAI.OneHtml.Css;
using RecursiveAI.OneHtml.Accessibility;
 
Document doc = Document.Parse(File.ReadAllBytes("events.html"));
var pageUrl = new Uri("https://library.example/events/");
 
// Links: resolve each href against the page address with System.Uri
foreach (Node a in doc.Root.QuerySelectorAll("a[href], area[href]"))
{
    var url = new Uri(pageUrl, a.GetAttribute("href")!);
    Console.WriteLine($"{url,-45} {a.TextContent!.Trim()}");
}
 
// Images: GetAttribute returns null when the attribute is absent, "" when it is empty
foreach (Node img in doc.Root.QuerySelectorAll("img[src]"))
{
    string? alt = img.GetAttribute("alt");
    Console.WriteLine($"{img.GetAttribute("src"),-22} alt: {(alt is null ? "(missing)" : $"\"{alt}\"")}");
}
 
// Every attribute of one element, in source order
Node first = doc.Root.QuerySelector("main img");
foreach (Attr attr in first.Attributes)
{
    Console.WriteLine($"{attr.QualifiedName} = {attr.Value}");
}
 
// Links as a screen reader lists them: the accessible name, not just the text
var tree = AccessibilityTree.Build(doc, DocumentStyles.Create(doc));
foreach (LinkEntry link in tree.CreateNavigationLists().Links)
{
    Console.WriteLine($"\"{link.Name}\" -> {link.Target}");
}
Text
https://library.example/                      
https://library.example/books/                Books
https://library.example/events/               Events
https://library.example/visit/                Visit
https://library.example/events/story-time/    Details
https://example.org/repair-cafe               Partner site
mailto:[email protected]                      [email protected]
/img/logo.svg          alt: "Harbor Street Library"
/img/story-time.jpg    alt: (missing)
/img/repair.jpg        alt: "Volunteers at a workbench fixing a lamp"
src = /img/story-time.jpg
width = 640
height = 360
"Harbor Street Library" -> /
"Books" -> /books/
"Events" -> /events/
"Visit" -> /visit/
"Details" -> /events/story-time/
"Partner site" -> https://example.org/repair-cafe
"[email protected]" -> mailto:[email protected]

Notes:

  • The library doesn’t resolve URLs for you here. System.Uri does it from the page address. If the page has a <base href>, read it first (doc.Root.QuerySelector("base[href]")) and resolve against that.
  • a[href], area[href] matches the elements a browser treats as links. An a without href is not a link.
  • GetAttribute returns the value with character references decoded. Attributes gives every attribute in source order, with QualifiedName, LocalName, Namespace and the source ranges of the name and value.
  • The first link has no text, so TextContent is empty. Its accessible name, from the navigation lists, is “Harbor Street Library”, taken from the image inside it. To know what a person hears for a link, use the accessible name.
  • For srcset, <picture> and <source>, query them the same way (img[srcset], picture source[srcset]) and split the attribute yourself. The library doesn’t parse srcset.

More about the tree and the navigation lists: The accessibility tree of a page.

Following links across pages: Web scraping in C#.