Extract URLs From Text

How it works

Two patterns run over the text: one for links with an explicit http or https scheme, and — when the option is on — one for bare www. links appearing in prose. Both capture greedily up to whitespace, a quote, or an angle bracket, which deliberately over-captures.

The interesting work is the repair pass. Written language wraps links in its own punctuation: a full stop because the link ended a sentence, a comma because it sat in a list, a paren because it was an aside. The repair loop peels these off the end one at a time — but a closing parenthesis is only removed while the URL holds more closers than openers. That single counting rule is what lets a wiki path ending in (v2) keep its paren while the sentence's own bracket falls away.

Stripping tracking parameters uses the standard URL parser (the same one browsers use) rather than string surgery: the link is parsed, the offending parameters deleted, and the URL reassembled. Links that cannot be parsed — the schemeless www. kind — are passed through untouched rather than run through a normalizer they never came from.

What counts as a tracking parameter

The utm_ family is an open convention, so it is matched by prefix — any parameter beginning utm_ goes. The rest is a fixed list of click identifiers attached by major platforms: fbclid, gclid, gbraid, wbraid, msclkid, twclid, yclid, mailchimp's mc_cid and mc_eid, and a few more. The bar for the list is strict: a parameter is only included when removing it cannot change the page the server returns. Anything ambiguous stays, because a link that stops working is a far worse outcome than one that still carries a tag.

When you'd use this

Collecting every source cited across a week of meeting notes. Auditing which domains a newsletter actually points to before forwarding it. De-noising share links before saving them somewhere permanent. Building the input file for a link checker from a page's HTML. Counting how many distinct domains a document leans on — the unique-domain stat answers that at a glance.

The visible text is not the destination

Extracting URLs from HTML finds the addresses in the markup, and those are the addresses that will actually be requested — which is not always what the page shows a reader.

A link can display one thing and go somewhere else entirely. <a href="http://elsewhere.example">bank.example.com</a> reads as a bank and leads anywhere. That mismatch is the mechanism behind most phishing, and it is invisible to anyone reading the rendered page rather than the source.

This is the main reason to extract from HTML rather than from copied text. The list you get is the set of places the document points, stated plainly, with the display text removed. Reviewing it takes a fraction of the time that hovering over every link does, and it catches the ones nobody hovers over.

Two things to look for. Domains that resemble a familiar one but are not it — character substitutions, an extra word, an unexpected suffix. And redirect or tracking wrappers, where the real destination sits in a query parameter; the URL decoder will show you what is inside.

Extraction is not a safety check. It tells you where the links go, and deciding whether that is acceptable is still yours.

Where a URL ends is a guess

There is no unambiguous rule for the end of a URL in running prose, so any extractor has to make a judgment, and two cases go wrong regularly.

Trailing punctuation is the common one. A sentence ending "see https://example.com/page." contains a full stop that is not part of the address — but a full stop is perfectly legal in a URL, so nothing in the character itself says which it is. Closing brackets and quotation marks have the same problem, and a URL that genuinely contains a bracket, as many wiki addresses do, is indistinguishable from one followed by a parenthesis.

The other case is a URL broken across lines. Text from a PDF or a hard-wrapped email splits long addresses at the margin, and the second half arrives as a separate line. What is extracted is the first fragment, which looks like a complete and valid URL and leads nowhere useful.

Both are visible if you look. Scan the extracted list for entries ending in punctuation and for entries that look truncated, and repair the source text before extracting rather than fixing the output. For hard-wrapped sources, run the text through the line-break tool first.