Datasets
Open Access
Web Data Commons - Hyperlink Graphs
- Citation Author(s):
- Submitted by:
- Alexander Outman
- Last updated:
- Mon, 08/03/2020 - 21:31
- DOI:
- 10.21227/H23S3B
- Data Format:
- Links:
- License:
- Creative Commons Attribution
1027 Views- Categories:
- Keywords:
Abstract
The graphs have been extracted from the 2012 and 2014 versions of the Common Crawl web corpera. The 2012 graph covers 3.5 billion web pages and 128 billion hyperlinks between these pages. To the best of our knowledge, the graph is the largest hyperlink graph that is available to the public outside companies such as Google, Yahoo, and Microsoft. The2014 graph covers 1.7 billion web pages connected by 64 billion hyperlinks. Below we provide instructions on how to download the graphs as well as basic statistics about their topology.
We hope that the graphs will be useful for researchers who develop
- search algorithms that rank results based on the hyperlinks between pages.
- SPAM detection methods which identity networks of web pages that are published in order to trick search engines.
- graph analysis algorithms and can use the hyperlink graphs for testing the scalability and performance of their tools.
- Web Science researchers who want to analyze the linking patterns within specific topical domains in order to identify the social mechanisms that govern these domains.
We also provide the page graph in the format expected by the WebGraph Framework developed by Sebastiano Vigna. The graph is represented using three files: .graph, .offsets, .properties. All three are necessary to load the network into the library.
Using the WebGraph Framework, which can be downloaded from Maven Central, these files can be loaded using the following line of code: BVGraph graph = BVGraph.loadMapped(baseName, new ProgressLogger()).
The extracted data is provided according the same terms of use, disclaimer of warranties and limitation of liabilities that apply to the Common Crawl corpus.
The Web Data Commons extraction framework can be used under the terms of the Apache Software License.
Dataset Files
Network Properties network.properties (1.47 kB)
Network Offsets network.offsets (4.57 GB)
Network Graph network.graph (52.86 GB)
example_index.txt (1.64 kB)- example_arcs.txt (821 bytes)
Open Access dataset files are accessible to all logged in users. Don't have a login? Create a free IEEE account. IEEE Membership is not required.
