资源描述
Bad Bot ReportBad Bots Strike BackREPORT | 20202 2020 Bad Bot Report Contents3 About the Bad Bot Report4 The Rebranding of Bad Bots Bad Bots as a ServiceThe Legality of Web ScrapingThe Rise of Mega Credential Stuffing AttacksBreaches and Loyalty Programs Nonprofits Suffering from Unscrupulous Bot Operators Every Industry Has a Different Bot ProblemBad Bots Increase Infrastructure CostsSocial Media Bad Bots in Elections Bad Bots Strike Back6 Understanding What Bad Bots Do8 Executive Summary of Findings 11 The Bad Bot LandscapeWhat is a Bad Bot?Bad Bot Sophistication LevelsBad Bots by IndustryBad Bot Sophistication by IndustryBad Bot Traffic by Website SizeBad Bot Identity: Impersonating ChromeBad Bots Are Still Growing OldBad Bots Going ResidentialAmazon Bad Bot Market Share Drops Mobile ISPs: A Specialized WeaponWhere Bad Bots OriginateRussia and China: The Most Blocked Countries26 Imperva Threat Research LabThreat ResearchIndustry Research 27 RecommendationsRecommendations for Detecting Bad Bot Activity29 About Imperva Application Security3 2020 Bad Bot Report About the Bad Bot ReportImpervas 2020 Bad Bot Report investigates the daily attacks that sneak past sensors and wreak havoc on websites. This is the 7th Annual Bad Bot Report. Its based on 2019 data collected from Impervas global network and includes hundreds of billions of bad bot requests anonymized over thousands of domains. Our goal is to offer guidance about the nature and impact of automated threats to those of you on the frontlines of website security. What makes this report unique is its focus on bad bot activity at the application layer (layer 7 of the OSI model). Automated application layer attacks differ from volumetric DDoS attacks, the latter of which manipulate lower-level network protocols.Bad bots interact with applications in the same way a legitimate user would, making them harder to prevent. They enable high-speed abuse, misuse, and attacks on your websites, mobile apps, and APIs. They allow bot operators, attackers, unsavory competitors, and fraudsters to perform a wide array of malicious activities.Such activities include web scraping, competitive data mining, personal and financial data harvesting, brute-force login, digital ad fraud, spam, transaction fraud, and more.4 2020 Bad Bot Report The Rebranding of Bad Bots Bad bots have long been the scourge of the internet. They lurk amidst real human traffic. Many businesses misunderstand the negative impacts of unfettered automated traffic. But others know that bad bots are not benign and have a very focused motivationto make money.Weve come a long way from the early days of basic ticket scalping bots. Today, on sites like ticketbots, you can purchase a sophisticated range of customized spinners, drop checkers, ticket downloaders, and pdf generators to purchase tickets to any event on any platform globally.To jump to the front of the line to buy limited-edition sneakers, it is easy to purchase any of the hypebots or sneaker bots available on websites like aiobots, hypebots, or anothernikebot.But the next evolution of bad bot development is already underway. Bad bots are trying to improve their image and appear legitimate. This new wave of bot operators are building businesses on their ability to scrape proprietary data from websites, package the data, and provide competitive data feeds to any company willing to purchaseall neatly packaged as “business intelligence” services.Bad Bots as a ServiceThis rebranding of “bad bots as a service” demonstrates itself in many ways. First, through the adoption of professional looking websites offering business intelligence services called pricing intelligence, alternative data for finance, or competitive insights. Typically, these businesses offer data products focused on specific industries. Second, there is increased pressure to purchase scraped data within your industry. No business wants to lose in the market place because the competition has access to data that is available to purchase. Finally, there is the growth of job postings looking for people to fill positions with titles like Web Data Extraction Specialist or Data Scraping Specialist. In this environment, it is difficult to see the bot problem disappearing any time soon.The Legality of Web ScrapingIn the most significant legal ruling in the ongoing HiQ vs Linkedin case1, the Ninth Circuit appellate court ruled in favor of allowing bots to scrape publicly available content. Linkedin is attempting to prevent the automated scraping of profiles by aggregator HiQ and is appealing the ruling. The litigation may yet end up in the United States Supreme Court2.1 Data Scraping Survives! (At Least for Now) Key Takeaways from 9th Circuit Ruling on the HIQ vs. Linkedin Case The National Law Review September 30, 20192 LinkedIn Files Petition to the Supreme Court in HiQ Web Scraping Case The National Law Review March 12, 2020Human Traffic Bad BotsWebsiteTraffic5 2020 Bad Bot Report The Rise of Mega Credential Stuffing AttacksBeyond content and price scraping, the biggest bad bot problem is credential stuffing and credential cracking. Every website with a login is subject to these attacks and a new phenomenon is emergingthe rise of mega credential stuffing attacks targeting one company. A recent attack that Imperva mitigated lasted 60 hours and included 44 million login attempts. In general, the availability of billions of breached credentials has helped fuel the rise in credential stuffing, but such large scale attacks can cause significant infrastructure strain leading to slowdowns or downtime. These large application layer credential stuffing attacks might be as damaging as volumetric DDoS attacks to any organization unprepared to handle such a high volume of bad bot requests. Breaches and Loyalty Programs With the continuing proliferation of customer loyalty programs online, credential stuffing attacks are increasingly lucrative. Inside every loyalty program account is some digital currency available to redeem or transfer to another account. The availability of credentials from data breaches combined with the growth of online loyalty programs is providing the perfect platform for bot operators to attack every e-commerce business. The rise of problems associated with account takeover is unfortunately inevitable. Nonprofits Suffering from Unscrupulous Bot Operators Stolen credit card numbers are a problem for more than the card owner. Credit card enumeration is run against website payment processors to determine if the credit card is validand non-profits suffer more than most. Bot operators enumerate through credit card numbers and make small donations to nonprofit organizations. If the donation is successful using a card number, the bot operator knows that the credit card is valid and can be used elsewhere to commit further fraud. But the problem for the nonprofit doesnt end there. The card owner will see the fraud on their account and complain to the credit card company. The credit card company now has to deal with chargebacks involving the non-profit. While the unsuspecting non-profit cannot afford to spend time and incur fees refunding such fraudulent donations.Every Industry Has a Different Bot ProblemWhile the goal of each bad bot operator might be different depending on their industry, bots are the tool of choice and are vital to their success. There is an ecosystem within many industries that rely on bots for survival. Without their use, many such operators would struggle to compete. In many cases, deploying bad bots is an essential business practice. Every industry has its own bad bot problem and ecosystem of bot operators. Some of these include: AIRLINES There is an ecosystem of online travel agents, aggregators, and competitors that use bots to scrape contentincluding flight information, pricing, and seat availabilitywhile criminals attempt to fraudulently access user accounts that contain loyalty program awards and credit card information3.A recent attack that Imperva mitigated lasted 60 hours and included 44 million login attempts6 2020 Bad Bot Report E-COMMERCE Competitors use bad bots to aggressively scrape pricing and inventory information. Grinchbots and Sneakerbots create denial of inventory problems for customers seeking limited edition items. Criminals use bad bots to commit fraud by stealing gift card balances and to access user accounts and credit card information4. EVENT TICKETING Brokers, scalpers, hospitality agencies, and corporations use bad bots to check for ticket availability and to purchase available seats to resell on secondary markets. Criminals access user accounts to steal tickets and credit card information5.Bad Bots Increase Infrastructure CostsFor any business whose website, mobile app, or API is the unfortunate target of malicious bots, they have to deal with more problems. Not only does it have to deal with the competitive pricing pressure resulting from the scraping bots, but it has to maintain infrastructure uptime and redundancy so that real customers arent inconvenienced. In addition, they also suffer from skewed decision-making metrics because their web traffic has been polluted by bad bots.Social Media Bad Bots in Elections Influencer bots are a tool used to spread propaganda. The role of influencer bots on social media will take center stage as the United States presidential election gets closer. Automated traffic launched by bot operators who remotely manage a vast number of aggregated social media accounts will aim to influence and change votes. Bad Bots Strike BackThe bot problem is real for every website and mobile app. Businesses have tried to protect themselves by adding bot protection capabilities to their solutions. But bot operators are expanding their operations and evolving into legitimate businesses. With increased financial resources, bot operators are also developing new methods to evade common bot detection techniques that ensure the arms race continues. Only recently have business leaders become savvy to what bad bots do. Many are incredulous about the scams being perpetrated. One thing is certain, with the rebranding of bot operations into business intelligence companies, the hiring of professional data extraction experts, and investment in new techniques to evade detection, bad bots will continue to strike back.3 How Bots Affect Airlines Imperva Threat Research4 How Bots Affect E-commerce Imperva Threat Research5 How Bots Affect Ticketing Imperva Threat ResearchBad bots will continue to strike back.7 2020 Bad Bot Report Understanding What Bad Bots DoBAD BOT PROBLEM HOW IT HURTS THE BUSINESS SIGNS YOU HAVE A PROBLEM INDUSTRIES TARGETEDPrice Scraping Competitors scrape your prices to beat you in the marketplace You lose business because your competitor wins the SEO search on price Lifetime value of customers worsens Declining conversion rates Your SEO rankings drop Unexplained website slowdowns and downtime, usually caused by aggressive scrapersAll businesses that show prices E-commerce Gambling Airlines TravelContent Scraping Proprietary content is your business. When others steal your content they are a parasite on your efforts Duplicate content damages your SEO rankings Your content appears on other sites Unexplained website slowdowns and downtime, usually caused by aggressive scrapersSimilar to price scraping, but in addition: Job boards Classifieds Marketplaces Finance TicketingAccount Takeover (a.k.a., Credential Stuffing, Credential Cracking) Stolen credentials tested on your site. If successful, the ramifications are account lockouts, financial fraud, and increased customer complaints affecting customer loyalty and future revenues Increase in failed logins Increase in customer account lockouts and customer service tickets Increase in fraud (lost loyalty points, stolen credit cards, unauthorized purchases) Increase in chargebacksAny business with a login page requiring username and passwordAccount Creation (a.k.a., Account Aggregation) Free accounts used to spam messages or amplify propaganda Exploit any new account promotion credits (money, points, free plays) Abnormal increases in new account creation Increased comment spam Drop in conversion rates of new accounts to paying customerMessaging platforms Social media Dating sites CommunitiesPromotion abuse Gambling Credit Card Fraud (a.k.a., Carding, Card Cracking) Criminals testing credit cards numbers to identify missing data (exp. date, CVV). Damages the fraud score of the business Increases customer service costs to process fraudulent chargebacks Rise in credit card fraud Increase in customer support calls Increased chargebacks processedAny site with a payment processor E-commerce Nonprofit/Charities Airlines Travel Ticketing Financial Gambling8 2020 Bad Bot Report BAD BOT PROBLEM HOW IT HURTS THE BUSINESS SIGNS YOU HAVE A PROBLEM INDUSTRIES TARGETEDDenial of Service Slows the website performance causing brownouts or downtime Lost revenue from unavailability of website Damaged customer reputation Abnormal and unexplained spikes in traffic on particular resources (login, signup, product pages, etc.) Increase in customer service complaintsAll industriesGift Card Balance Checking Steal money from gift card accounts that contain a balance Poor customer reputation and loss of future sales Spike in requests to the gift card balance page. Increase in customer service calls about lost balancesE-commerceDenial of Inventory Bots hold items in shopping carts, preventing access by valid customers Damaged customer reputation because unscrupulous middle men hold all inventory until resold elsewhere Increase in abandoned items held in shopping carts Decrease in conversion rate Increase in customer service calls about lack of availability of inventoryScarce or time-sensitive items Airlines Tickets E-commerce (Sneakers)9 2020 Bad Bot Report Executive Summary of Findings Bad Bot Traffic Rises to Highest EverIn 2019, bad bot traffic rose to its highest ever percentage of 24.1 percent of all traffic. 37.2 percent of all internet traffic wasnt human. Human traffic increased by 1.1 percent to 62.8 percent of all traffic.Bad Bot Sophistication Levels Remain Consistent for the Third YearAdvanced persistent bots (APBs) continue to plague websites and often avoid detection. APBs cycle through random IP addresses, enter through anonymous proxies, change their identities, and mimic human behavior.Good Bots Traf_fic Percentage in 2019 13.1%25.1%Percentage change in good traffic from previous yearBad Bot v Good Bot v Human Traf_fic 2019Bad Bots Amount all Website Traf_fic in 2019 24.1%18.1%Percentage change in bad bot traffic from previous year24.1%Bad Bots62.8%13.1%Good BotsHuman1.1%Human Website Traf_fic Percentage in 2019 62.8%Percentage change in human traff
展开阅读全文