Into the Dark: Unveiling Internal Site Search Abused for Black Hat SEO

Into the Dark: Unveiling Internal Site Search Abused for Black Hat SEO

Speakers: Yiming Zhang, Tsinghua University; Yunyi Zhang, Mingxuan Liu, Baojun Liu, Haixin Duan, Min Zhang, Hui Jiang, Baidu Inc; Yanzhe Li, Fan Shi

Conference: USENIX Security '24

YouTube: https://www.youtube.com/watch?v=cmVL1wzxiTU

Overview

This talk, presented by Yiming Zhang from Tsinghua University and a collaborative effort with researchers from Baidu Inc., introduces and thoroughly investigates a novel black hat SEO technique termed Internal Site Search Abuse for Promotion (ISAP). ISAP exploits the internal search functionality of high-reputation websites to inject and promote illegal services, such as online gambling or adult content, within legitimate search engine results. The researchers highlight ISAP's low cost, high effectiveness, and widespread impact, revealing that it does not require domain registration or website compromise, yet successfully manipulates search engine rankings across major platforms like Baidu, Google, and Bing.

The presentation delves into the mechanics of ISAP, explaining how malicious actors leverage the often-unsecured implementation of internal site search functions to generate indexed URLs containing promotional content. A key contribution of this work is the development and deployment of a lightweight, two-stage detection scheme, incorporating a URL screener and a BERT-based semantic detector, which has been successfully implemented by Baidu. The findings from this large-scale deployment are alarming, identifying millions of abusive URLs and demonstrating significant real-world impact with millions of user clicks.

Beyond detection, the researchers also conducted a systematic measurement study to uncover the strategies employed by ISAP promoters and evaluated the broader security risks. They propose practical mitigation measures for website administrators and search engine providers, emphasizing the urgent need for a unified standard for internal search implementation. This research not only sheds light on a previously under-explored vector for black hat SEO but also provides concrete solutions and insights for defending against this pervasive threat.

Background

▶ Watch: Introduction to ISAP: A new Black Hat SEO technique (0:00)

The digital landscape is constantly battling the clandestine promotion of illegal services. Traditional methods employed by malicious actors, collectively known as black hat SEO, typically involve techniques like keyword stuffing to inflate relevance or link farms to artificially boost page authority. However, these methods often require significant investment in domain registration, content creation, or even compromising existing websites. The talk reveals a new, more insidious approach that circumvents these traditional barriers: Internal Site Search Abuse for Promotion (ISAP).

The fundamental premise of ISAP hinges on the common yet often overlooked feature of internal site search. Many websites, particularly those with extensive content like universities, news portals, or government agencies, provide a search box to help users quickly locate specific information within their domain. A user might type "scholarship" into a university's internal search, expecting to see a list of relevant sub-pages. The vulnerability arises from how these internal search functions are implemented. Crucially, there is no unified standard for how websites should handle internal search queries. This lack of standardization leads many sites to generate new sub-pages where both the URL and the content of the result page dynamically incorporate the user's search query, often by using HTTP GET requests that embed the keyword as a URL parameter.

This seemingly innocuous functionality becomes a potent weapon in the hands of black hat SEO promoters. By injecting their promotional content—disguised as a search keyword—into a high-reputation website's internal search, they can force the website to generate a unique URL under its legitimate domain. This newly generated URL, containing the illegal promotional content, then benefits from the inherent trust and authority of the host domain. When search engines crawl and index these "reflection URLs," they appear as highly authoritative results, potentially ranking high in normal user searches. The low cost—requiring no domain registration or website compromise—and the leverage of existing high-reputation infrastructure make ISAP an attractive and effective technique for promoting illicit services, posing a significant challenge to both search engines and website administrators.

Key Findings

▶ Watch: Step-by-step: The ISAP attack process (3:00)

The research uncovered several critical findings regarding the prevalence, impact, and operational strategies of ISAP, highlighting its significant threat landscape:

• Widespread Abuse and Impact: The most striking finding is the sheer scale of ISAP abuse. Through a five-month detection period using their proposed scheme, Baidu identified over 3 million abusive URLs and confirmed that over 10,000 popular websites had been impacted. The real-world consequence is substantial: user click data from Baidu revealed that these identified abuse URLs were clicked by more than 6 million users in just four days, underscoring the immediate and pervasive threat ISAP poses to internet users.
• Cross-Search Engine Effectiveness: ISAP is not confined to a single search engine. The researchers sampled detected URLs from Baidu and verified their promotional effectiveness on Google and Bing. They found that these ISAP URLs were indeed indexed by both search engines, and manual searches for relevant keywords confirmed their exposure to users, with some even appearing on the first page of search results. This demonstrates that ISAP successfully exploits fundamental search engine indexing and ranking mechanisms, making it a cross-platform threat.
• High Vulnerability Rate: A systematic scan for vulnerable websites across various domain lists, including top-level, educational (.edu), and government (.gov) domains, revealed a concerning vulnerability rate. On average, over 18% of these high-reputation websites were found to be susceptible to ISAP, indicating a systemic flaw in internal search implementation across a broad spectrum of critical online infrastructure.
• Preferred Illegal Services: The study identified specific types of illegal services that ISAP promoters preferentially target. The primary categories include gambling and adult content (OT) services. Interestingly, the researchers also observed a growing trend where ISAP is used to promote black hat SEO services themselves, creating a self-perpetuating cycle of abuse.
• Promotion Target Modalities: Promoters mainly embed their contact information or access points as the promotion target within the keywords. These targets predominantly include domain names (e.g., for illegal gambling sites), Telegram or WeChat accounts (for direct communication), and telephone numbers. These are combined with popular "hot keywords" to maximize visibility.
• Sophisticated Promoter Strategies:
• Keyword Length: ISAP promoters craft their keywords strategically. They combine the promotion target with frequently searched "hot keywords" (e.g., "Sun City," a famous gambling venue in Macau). This results in longer, more specific search queries. The analysis of ground truth data showed that over 90% of promotion keywords exceeded 14 characters in length, differentiating them from typical benign search queries.
• Distribution Websites for Indexing: To ensure the reflection URLs are indexed by search engines, promoters utilize "distribution websites." These are legitimate websites that are already indexed by search engines but allow the addition of new external links, such as blog comments or forum posts. By embedding the reflection URLs as external links on these distribution sites, promoters guide search engine crawlers to discover and index the ISAP-generated content. These distribution websites were observed to have a higher number of external links compared to benign ones, suggesting a deliberate strategy to enhance indexing effectiveness.

These findings collectively paint a clear picture of ISAP as a significant and evolving threat, requiring concerted efforts from search engines, website administrators, and security researchers for effective mitigation.

Technical Deep Dive

▶ Watch: Empirical study: Observations from ground truth data (5:00)

The ISAP technique is characterized by a series of precise steps that exploit specific vulnerabilities in internal site search implementations and search engine indexing mechanisms. The researchers meticulously detailed this attack chain and subsequently developed a robust detection scheme.

ISAP Attack Chain

The attack unfolds in a structured, multi-stage process:

1. Promotion Target Selection: The attacker first identifies an illegal service or product they wish to promote, such as an online gambling platform. They then determine the specific information to be promoted, which could be a domain name, a Telegram ID, or a phone number.
2. Promotion Keyword Generation: Next, the attacker crafts a promotion keyword. This keyword is a sophisticated blend of the chosen promotion target and a highly popular or "hot" keyword that users frequently search for. For instance, to promote a gambling site, the attacker might combine the site's URL with "Sun City," a well-known gambling venue. This strategy ensures that when a legitimate user searches for the hot keyword, the ISAP-generated content might appear, and the promotion target is subtly embedded. The length of these combined keywords is often notably longer than typical user queries, with over 90% exceeding 14 characters in the observed dataset.
3. Vulnerable Website Identification: Promoters actively seek out high-reputation websites that are vulnerable to ISAP. A website is considered vulnerable if its internal search function generates a new sub-page where both the URL and the content of the result page dynamically include the user's search query, even if the keyword itself doesn't exist within the site's content. This often occurs when the internal search uses HTTP GET requests, embedding the keyword as a parameter in the URL.
4. Reflection URL Generation: Once a vulnerable website is identified, the attacker uses its internal search function to search for their carefully crafted promotion keyword. The vulnerable website, in response, generates a unique sub-page. This sub-page's URL and/or content will contain the promotion keyword, effectively creating a reflection URL under the legitimate, high-reputation domain.
5. Distribution Website Utilization: To ensure these reflection URLs are discovered and indexed by search engines, attackers employ distribution websites. These are typically legitimate sites (e.g., blogs, forums, news comment sections) that are already regularly crawled by search engines and allow users to post external links. The attackers embed the generated reflection URLs as external links on these distribution websites. This acts as a signal to search engine crawlers, guiding them to the newly created ISAP URLs.
6. Search Engine Indexing: Search engine crawlers follow these external links from the distribution websites, discover the reflection URLs, and proceed to index them. Due to the high reputation of the host domain (e.g., a university or government site), these indexed ISAP URLs are often assigned a high ranking.
7. User Exposure: When a normal user searches for the "hot keyword" (e.g., "Sun City") on a major search engine, the highly ranked reflection URL appears in the search results. The user sees the promotional content, and the ISAP attack is successful.

Lightweight Detection Scheme

To combat ISAP, the researchers developed and implemented a lightweight, two-stage detection scheme, specifically designed to process billions of daily search traffic data from a search engine in just two hours. This efficiency is crucial for real-time threat mitigation.

1. URL Screener: This is the first stage of the detector, designed to drastically reduce the volume of data requiring deeper analysis. It operates based on empirical features derived from a ground truth dataset provided by Baidu's security department, which contained both legitimate and user-reported ISAP abuse cases.

• Filtering Criteria: The screener employs rules to filter out unlikely ISAP candidates. For example, it filters out URLs whose associated distribution websites have a low number of external links (as ISAP promoters prefer highly linked distribution sites). It also filters out URLs generated from very short search keywords, based on the observation that over 90% of ISAP promotion keywords exceed 14 characters.
• Data Reduction: This stage is highly effective, reducing the scale of data to be analyzed from 60 million daily URLs to just 10,000, making subsequent, more resource-intensive analysis feasible.

1. Semantic Detector: The second stage is a more sophisticated model built on BERT (Bidirectional Encoder Representations from Transformers), a powerful pre-trained language model for natural language processing.

• ISAP vs. Benign Classification: The BERT-based model performs a semantic analysis of the keywords embedded in the URLs. It is trained to differentiate between ISAP URLs and benign ones by understanding the contextual meaning and intent behind the keyword combinations.
• Business Classification: Beyond mere detection, the semantic detector can further classify the specific types of illegal businesses being promoted (e.g., gambling, adult content, black hat SEO services) based on the semantic patterns observed in the keywords. This provides valuable intelligence for understanding promoter trends and for targeted remediation efforts.
• Performance: This detector achieved good performance on the labeled ground truth dataset, demonstrating its accuracy in identifying and categorizing ISAP instances.

Vulnerability Scanning Method

To identify potentially vulnerable websites proactively, the researchers devised a systematic scanning method:

1. Search Box Enumeration: The first step involves enumerating all possible search input fields on a given website. This is done by analyzing the website's HTML structure to identify common search box elements.
2. Internal Search Function Identification: Using rule-based methods, the scanner then distinguishes genuine internal site search functions from other search forms (e.g., external search widgets, login forms). This involves analyzing form actions, input names, and surrounding HTML context.
3. Vulnerability Check: For each identified internal search function, the scanner performs a series of tests. It searches for a set of carefully chosen keywords, including non-existent ones.

• Vulnerability Criteria: A website is deemed vulnerable if, upon searching a keyword, it generates a new sub-page where both the URL and the displayed web page content contain the searched keyword. This behavior, especially for non-existent keywords, is a clear indicator of the exploitable dynamic page generation.

This comprehensive technical approach, encompassing both a detailed understanding of the attack and a sophisticated, deployable detection and scanning framework, forms the core of the research's contribution.

Demo / Proof of Concept

▶ Watch: Real-world impact: Millions of abuses, services, user clicks (7:00)

While the talk did not feature a live, real-time hacking demonstration, the researchers provided compelling evidence and a practical "proof of effectiveness" for ISAP's impact across major search engines. Their demonstration focused on validating that the ISAP URLs detected by Baidu were indeed indexed and actively promoting illegal content on other prominent search platforms.

The process involved:

1. Sampling Detected URLs: The team took a subset of the millions of ISAP abuse URLs that their detector had identified and removed from Baidu's index.
2. Cross-Platform Verification: They then manually searched for the relevant "hot keywords" and promotion targets associated with these sampled URLs on Google and Bing.
3. Observation of Exposure: The results confirmed that the same ISAP URLs were also being indexed by Google and Bing. Critically, when searching for the corresponding keywords, these abusive URLs were found to be exposed to users in the search results, with some instances even appearing on the first page of the search results.

This systematic verification process served as a powerful demonstration of ISAP's pervasive nature and its ability to bypass the defenses of multiple search engines. It underscored the real-world impact and confirmed that the threat extends far beyond a single platform, solidifying the need for broader industry-wide awareness and mitigation. The demonstration effectively transitioned from detection on one platform to validating the widespread effectiveness of the attack technique across the global search landscape.

Defensive Implications

▶ Watch: ISAP's prevalence across Google and Bing (8:10)

The findings of this research provide crucial insights for both website administrators and search engine providers to defend against ISAP. Effective mitigation requires a multi-faceted approach, addressing both the root cause of the vulnerability and the detection of its exploitation.

For Website Owners and Administrators:

The primary responsibility for preventing ISAP lies with website owners, particularly those managing high-reputation domains. The proposed mitigations target the specific implementation flaws in internal site search functions:

1. Use HTTP POST for Internal Site Search: The core vulnerability often stems from internal search functions using HTTP GET requests. When HTTP GET is used, the search keyword is appended directly to the URL as a query parameter (e.g., example.com/search?q=keyword). This allows the website to generate a new URL containing the injected promotional content, which can then be indexed. By switching to HTTP POST for internal searches, the search keyword is sent in the body of the HTTP request, not in the URL. This prevents the generation of unique, indexable URLs containing the attacker's keyword, thereby eliminating the primary vector for ISAP.
2. Return 404 for Non-Existent Keywords: Many vulnerable websites, even if they don't find content for a searched keyword, still generate a result page (often an "empty results" page) and crucially, include the searched keyword in the URL or the page title/content. This behavior is exploitable. Instead, if an internal search query yields no results, the website should return a 404 Not Found HTTP status code. This clearly signals to search engines that the requested resource (the manipulated search result page) does not exist, preventing its indexing and disassociating the domain's reputation from the malicious content. It also discourages promoters as their generated URLs would quickly be de-indexed.
3. Regular Audits and Monitoring: Website administrators should regularly audit their internal search functionalities to identify and rectify any vulnerabilities. Tools and methods similar to the scanning technique proposed in this research can be employed to proactively test for ISAP susceptibility. Monitoring internal search logs for unusually long or suspicious keyword patterns could also provide early warning signs.

For Search Engine Providers:

Search engines play a critical role in detecting and neutralizing ISAP attacks that have already bypassed website-level defenses:

1. Implement Robust Detection Schemes: Search engines should implement sophisticated detection schemes, such as the lightweight, two-stage detector proposed in this research. The combination of a URL Screener to efficiently filter massive data and a BERT-based Semantic Detector for accurate classification has proven effective. This allows for the timely identification of ISAP URLs within their vast indexes.
2. Regular Removal and De-indexing: Upon detection, ISAP URLs must be promptly and regularly removed or de-indexed from search results. Baidu's successful implementation, which regularly removes these URLs, demonstrates the effectiveness of this proactive approach in mitigating user exposure. Bing has also reportedly made fixes based on this research.
3. Algorithm Adjustments: Search engine ranking algorithms could be refined to de-prioritize or penalize URLs that exhibit characteristics consistent with ISAP, such as dynamic pages with unusual keyword combinations originating from internal search functions, especially if linked from low-reputation distribution sites.
4. Collaboration and Information Sharing: Search engines should actively collaborate with security researchers and share threat intelligence to stay ahead of evolving black hat SEO techniques. Furthermore, they should establish clear channels to report vulnerable websites to their respective owners for remediation. The researchers themselves contacted 50 organizations with vulnerable websites, and 37 of them have already fixed the problem, underscoring the importance of this outreach.

By adopting these defensive measures, the internet community can collectively reduce the effectiveness of ISAP, protect users from illegal content, and preserve the integrity of high-reputation websites.

Key Takeaways

• ISAP is a Novel and Pervasive Threat: Internal Site Search Abuse for Promotion (ISAP) is a new, low-cost, and highly effective black hat SEO technique that leverages the internal search functions of high-reputation websites to promote illegal services.
• Leverages High-Reputation Domains: The technique exploits the trust and authority of legitimate domains (e.g., universities, government sites) to achieve high search engine rankings for illicit content, affecting major search engines like Baidu, Google, and Bing.
• Sophisticated Detection is Possible: A lightweight, two-stage detection scheme, combining a URL screener for efficient data reduction and a BERT-based semantic detector for accurate classification, has proven effective in identifying millions of ISAP URLs.
• Significant Real-World Impact: ISAP has abused over 10,000 popular websites, resulted in 3 million detected abusive URLs, and led to over 6 million user clicks on illegal content in just four days, highlighting its substantial real-world impact.
• Clear Mitigation Strategies Exist: Website owners can prevent ISAP by implementing internal site search using HTTP POST requests and by returning a 404 error code for non-existent search keywords instead of generating dynamic result pages.
• Collaboration is Key for Remediation: Ongoing efforts by researchers, search engine providers, and website administrators to detect, remove, and fix vulnerabilities are crucial for combating ISAP and protecting the integrity of online information.

About the Speaker(s)

The talk was presented by Yiming Zhang from Tsinghua University. This research was a collaborative effort involving a team of researchers from Tsinghua University and Baidu Inc., including Yunyi Zhang, Mingxuan Liu, Baojun Liu, Yiming Zhang, Haixin Duan, Min Zhang, Hui Jiang, Yanzhe Li, and Fan Shi. The presenter, Yiming Zhang, shared the team's extensive work on uncovering, analyzing, and mitigating the novel black hat SEO technique known as ISAP. Their collaboration with Baidu Inc., a leading search company, provided critical real-world data and a platform for implementing their detection scheme, demonstrating a strong link between academic research and practical industry application in cybersecurity.