CommonCrawl

resource thumbnail

Remove from Bookmarks

Do you really want to remove?
This action cannot be undone. Choose 'Cancel' to stop and go back.
Cancel
Ratings: 0
  • Which text to add here??

Added by benbanbun on 2011-11-07 08:52

» Viewed 554 times
» Favorited by 0 user(s)
» 0 Comments
» This resource has public visibility

Holder of Rights:

License: unknown

Creator(s):

Description:
Common Crawl produces and maintains a repository of web crawl data that is openly accessible to everyone. The crawl currently covers 5 billion pages and the repository includes valuable metadata. The crawl data is stored by Amazon’s S3 service, allowing it to be bulk downloaded as well as directly accessed for map-reduce processing in EC2. This makes wholesale extraction, transformation, and analysis of web data cheap and easy. Small startups or even individuals can now access high quality crawl data that was previously only available to large search engine corporations.

Add to Collection

You don't have any collections yet. Click here to create your first collection!

Okay

Share to Group

You don't have any group you can share this resource with: the resource is already shared to all groups you are member in. Click here to see available groups!

Okay

Create QR Code

Please select the URI for the QR Code:




Create Cancel

Tags

sort: alphabetically • by frequency
use blanks to separate tags

Comments

CommonCrawl Common Crawl produces and maintains a repository of web crawl data that is openly accessible to everyone. The crawl currently covers 5 billion pages and the repository includes valuable metadata. The crawl data is stored by Amazon’s S3 service, allowing it to be bulk downloaded as well as directly accessed for map-reduce processing in EC2. This makes wholesale extraction, transformation, and analysis of web data cheap and easy. Small startups or even individuals can now access high quality crawl data that was previously only available to large search engine corporations.