Social Media marketing Impact: Common trends on social media can spill more than into engines like google, influencing the research quantity of related keyword phrases. txt file is then parsed and may instruct the robot as to which webpages aren't to become crawled. As being a online search engine crawler https://www.youtube.com/watch?v=RSlwOZLlUGg