WP Content Crawler – Get content from almost any site, automatically!

WP Content Crawler – Get content from almost any site, automatically! Created by turgutsaricam. This templates have specific characteristics, namely: Get content from almost any site to your WordPress blog, .

You can use this WP Content Crawler – Get content from almost any site, automatically! on WordPress category for autoblog, autoblogging, automatic, automatically, bot, content, crawl, crawler, post, product, programmatically, save, saver, scheduling, woocommerce and another uses.

WP Content Crawler - Get content from almost any site, automatically!
© Copyright by turgutsaricam

Main Features:

  • Video preview resolution: None
  • Software version: WordPress 4.6, WordPress 4.5.x, WordPress 4.5.2, WordPress 4.5.1, WordPress 4.5, WordPress 4.4.2, WordPress 4.4.1, WordPress 4.4, WordPress 4.3.1, WordPress 4.3, WordPress 4.2, WordPress 4.1, WordPress 4.0, WordPress 3.9
  • Compatible with: None
  • Compatible browsers: IE9, IE10, IE11, Firefox, Safari, Opera, Chrome, Edge
  • High resolution: None
  • Files included: JavaScript JS, CSS, PHP

Get content from almost any site to your WordPress blog, automatically!

For what it can be used

  • Create a personal site which collects news, posts, etc. from your favorite sites to see them in one place
  • Use it with WooCommerce to collect products from shopping sites
  • Collect products from affiliate programs to make money
  • Collect posts to create a test environment for your plugin/theme
  • Collect plugins, themes, apps, images from other sites to create a collection of them
  • Keep track of competitors

How it works

It’s all about CSS selectors and you can learn how to use them in minutes by watching the introduction tutorial.

See it in action, learn in minutes

Available in two languages, ready for translation

Video tutorials



Requirements:

PHP >= 5.5, mbstring

Tested with WP versions:

4.6.1, 4.5.3, 4.4.2, 4.3.3, 4.2.7, 4.1.10, 4.0.10, 3.9.11

Main features:

  • Save post’s title, excerpt and content.
  • Save meta keywords and description.
  • Save images in the posts to your site.
  • Save featured image of target post as the featured image of the post created in your site.
  • Save the list in posts (extract the list from the post).
  • Save paginated posts. If target post is paginated, the post in your site can be paginated as well.
  • Save anything from target page as post meta.
  • Find and replace anything in the target page’s HTML before saving its content.
  • Find and replace using regular expressions.
  • Find and replace in image URLs before saving them.
  • Manipulate target page’s HTML before saving to make it suitable for your needs.
  • Remove unnecessary elements from target page’s HTML.
  • Map categories of your site to target site’s categories to keep posts organized.
  • Automatically get category URLs for category mapping.
  • Add unlimited categories for a site and automatically check for new posts uniformly.
  • Set alternative CSS selectors for each setting. By this way, get content from the pages of the same site which are designed differently.
  • Automatically save posts.
  • Automatically check for new posts.
  • Set maximum number of category pages to be checked automatically.
  • Add unlimited sites.
  • Import/export site settings.
  • Set post type. For instance, you can add posts as products if you use WooCommerce.
  • Set a template for the post.
  • Set a template for each list item.
  • Set passwords for created posts.
  • Keep notes for each site to keep a changelog or anything you want.
  • Test any CSS selector and find-and-replace setting right away.
  • Test a site before activating it for automatic crawling.
  • Manually create posts by providing the URL of the posts.
  • If you want, use settings different than general settings for each site.
  • Activate/deactive each site any time to start/stop automatic crawling.
  • Either directly publish the posts or keep them as drafts to make changes before publishing.
  • Online documentation
  • Video tutorials for a quick start
  • Ready to translate (.po file)
  • Get updates from your WordPress admin panel.
  • Save images as gallery. If you use WooCommerce, you can save images from target page as product gallery.
  • Get anything from target page as a shortcode and use the shortcodes in templates to place any data anywhere you want.
  • Save tags using CSS selectors.
  • Find and replace in post tags.
  • Limit the number of tags that can be added to a post.
  • Use a proxy or proxies to get content from the sites to which your IP does not have access.
  • Post title and excerpt templates in which you can use custom short codes.
  • Find and replace in custom short code data.
  • Add custom post meta without a selector.
  • Set how many times URL collection and post crawling events should run each time for a site. For instance, you can save 3 posts every minute, or run URL collection 5 times every 2 minutes.
  • Collect post URLs in reverse order for each category page.
  • Remove links from the content by just checking a checkbox. This will not touch the links manually added to the templates.
  • Notifications. Set CSS selectors whose values should not be empty for category and post pages. When an empty value is found using those selectors, you can get an email notification.

Changelog

v1.2.1 - 12 October 2016* Improvement: Selectors for images that should be saved are searched in the short code data as well.* Improvement: Partial French translation is added, thanks to @PECNET* Fixed: JS and CSS files are not updated after the plugin is updated. (Adds versions to CSS and JS files)* Fixed: Template unnecessary element selectors do not work.* Fixed: When a relative URL does not start with a forward slash, the final URL should be prepared relative to the current URL.v1.2.0 - 30 August 2016* New: You can now use proxy.* New: Set connection timeout in seconds.* New: Post title and excerpt templates in which you can use custom short codes.* New: Find and replace in custom short code data.* New: Maximum number of categories that can be added automatically via CSS selectors to the category map increased.* New: Add custom post meta without a selector.* New: You can set how many times URL collection and post crawling events should run each time for a site. For instance, you can save 3 posts every minute, or run URL collection 5 times every 2 minutes.* New: You can collect post URLs in reverse order for each category page.* New: Remove links from all short code data. This will not touch the links manually added to the templates.* New: Notifications. You can now set CSS selectors whose values should not be empty for category and post pages. When an empty value is found using those selectors, you can get an email notification.* Fixed: Downloaded file's name does not have a proper file extension if the file on the target site is generated dynamically.* Fixed: Crawling stops if there is a request exception.* Fixed: Crawling stops if target page's HTML could not be retrieved.v1.1.3 - 26 June 2016* Fixed: Fetching stops if there is a URL that cannot be fetched.* Fixed: JS error saying that sortable is not a function.* Fixed: Tag selector results are taken as single results. Make it multiple and get all of the results found by each selector.* Fixed: Storage directory's CHMOD should be 755 to prevent any security breaches.* Improvement: License key input field is now shown as password.* Improvement: Each post will now have _wpcc_post_url post meta that stores the URL of the post.* Improvement: If you leave any URL in category map empty, you will get an error.* Improvement: Now you will get a notification when your PHP version is lower than 5.5.* Improvement: Now you can safely deactivate plugin from license key settings page as well, which also removes license key from your current domain and makes it available to be used on other domains.v1.1.2 - 21 May 2016* Fixed: Crawler stops saving posts although there are URLs in the queue.* Fixed: Crawler stops crawling if there is no category URL in an active site.v1.1.1 - 16 May 2016* New Feature: Save tags using CSS selectors* New Feature: Find and replace in post tags* New Feature: Limit the number of tags that can be added to a post* Fixed: Export-settings field should not be send with the request.* Fixed: 'Failed to open stream for upgrade.php' error caused by DatabaseService.php* Fixed: Category list causes an error when WooCommerce is not installed.v1.1.0 - 15 May 2016* New Feature: Create custom short codes for CSS selectors and place any element or attribute of any element at anywhere in the post template.* New Feature: Remove unnecessary elements after the post template is ready.* New Feature: Add images as a gallery. You can also add image gallery as a WooCommerce product's image gallery.* Improvement: Regex errors are now shown at details section in Tester page.* Improvement: Content existence is controlled in a smarter way.* Improvement: You can now use your license on another site after you deactivate the plugin on the site which the plugin is already installed.* Fixed: Firefox does not let copying texts from disabled textareas.* Fixed: Quick-edit deletes all settings.* Fixed: Select inputs in custom general settings are not assigned their value from database.* Fixed: Export-settings textarea should be 'readonly'.* Fixed: Hard-coded directories should be defined dynamically.* Fixed: CHMOD of storage directory should be set as 777.* Fixed: The results of post content selectors are not ordered by the positions of the found elements.* Fixed: Regular expressions are not imported appropriately when importing settings.* Fixed: Regex errors cause whole template to be empty.* Fixed: Template editors remove HTML tags.* Fixed: Category of the post is not assigned as selected when saving a URL more than once using manual crawling.* Fixed: Database tables are not created on multisite.* Fixed: Some sites are not available to select in tools page.v1.0.1 - 9 May 2016* Fixed: MySQL does not let thumbnail_url be null.