URL Clustering for Piracy Investigations

Raw piracy monitoring produces URLs, and URLs exaggerate. The same episode on the same file host might appear as a direct link, a link with an affiliate parameter, a mobile-subdomain version, a shortened link on social media and copies on mirror domains that proxy the original site. Counted separately, that looks like a large problem. Handled separately, it means verifying the same file again and again and sending a platform a pile of notices that all describe one object. URL clustering fixes both problems.

Normalise before comparing

Clustering only works on clean input. Normalisation removes what does not identify content: tracking and session parameters, fragments, trailing slashes, upper-case hostnames, "www" and mobile prefixes. Shortened links and redirectors are expanded to their final destination, and the redirect chain is kept as evidence of how a user would have reached it. Internationalised domain names are converted to a single representation so lookalike characters do not split what is really one site.

After this step, a good share of detections collapse into exact duplicates and are merged without any judgement. The real work begins with URLs that differ but are related.

What holds a cluster together

Several features are used, usually in combination:

• Content identifiers in the path. File hosts and video platforms put a file or video ID in the URL. Two URLs with the same ID on the same host, or on a host and its mirror site, point to the same object.
• Final destination. Aggregator pages whose own URLs share nothing may all resolve to the same embed or file. They belong to one delivery cluster.
• Path patterns. Pirate sites are built from templates, so episode pages follow a predictable structure, typically a show slug followed by season and episode segments. Recognising the pattern groups every episode page of a series on one site and helps predict where new episodes will appear.
• Response fingerprint. The same page title format, error page and script bundle across several domains suggests mirrors of one site.
• Verified content. Where an analyst has confirmed what each URL actually plays, the verified title and episode is the strongest grouping key of all.

Different questions call for different keys. "Everywhere this episode is available" groups by content. "Everything this host needs to remove" groups by the responsible platform. "Every page belonging to this site and its mirrors" groups by site. A mature system keeps all three views over the same detections rather than forcing one.

Clusters as units of work

The operational payoff lies in how work is organised. Verification happens per cluster: an analyst confirms the content behind a representative URL, checks that the others genuinely resolve to the same object, and records the decision for all of them. Spot checks within large clusters guard against a mislabelled member slipping through.

Notices are assembled per responsible party. A file host receives one notice listing every URL it controls for a title, rather than many fragments arriving over a day. That is easier for the platform to process and tends to be handled faster. Search engine delisting is the exception, since each URL is a separate entry in the index and has to be listed individually, but even there clustering prevents the same URL from being submitted twice.

Reporting changes too. A client can see how many distinct copies exist, how many sites carry them and which platforms are responsible, alongside the raw link count. Both numbers are honest; they answer different questions, and confusing them leads to poor decisions about where to spend effort.

When to split a cluster

Clusters are working hypotheses. A shared host is not shared content: a single cyberlocker stores files from countless uploaders. A shared template is not a shared operator: pirate site scripts are sold, cloned and reused. Grouping by these features is reasonable for routing work and unreasonable as a basis for saying two sites are run by the same people. When verified content differs, or a URL turns out to be an authorised copy on the client's whitelist, it is split out with a note explaining why.

That discipline is what lets clustering speed up an investigation without introducing errors into notices. It also connects to wider analysis, where the question shifts from which URLs are the same item to who stands behind them, a job for OSINT investigation.

DigiGuardians groups and verifies detections before enforcement so that notices act on the source as well as the link, and the client's dashboard shows what was found, where, and what happened to it. The service is described under content protection.