Skip to main content

​
Problem

Dynamic filtering components (like the Contract Addresses Table and DVN Addresses Table) update URLs with query parameters to make filter states shareable: 
/deployments/deployed-contracts?chains=ethereum,bsc&stages=mainnet
/deployments/dvn-addresses?dvns=layerzero&chains=polygon
Search indexers like Typesense treat each unique URL as a separate page, creating hundreds of duplicate content entries for the same base page content.

​
Solution

We implement a multi-layered approach to prevent duplicate content indexing:

​
1. Meta Tags (Primary Solution)

The useSearchIndexing hook automatically adds appropriate meta tags when filters are active: 
// When filters are active:
<link rel="canonical" href="/deployments/deployed-contracts" />
<meta name="robots" content="noindex, follow" />
How it works:
  • Canonical URL: Points to the base page without query parameters
  • Noindex: Prevents indexing of filtered pages while allowing link following

​
2. Robots.txt (Secondary Protection)

Additional robots.txt rules block crawlers from accessing filtered URLs: 
# Block crawling of filtered URLs
Disallow: /*?*chains=*
Disallow: /*?*dvns=*
Disallow: /*?*stages=*
Disallow: /*?*page=*

​
3. Usage in Components

For any component with filtering, use the hook: 
import {useSearchIndexing} from '../hooks/useSearchIndexing';

function MyFilterableComponent() {
  const [filters, setFilters] = useState([]);

  // Determine if any filters are active
  const hasActiveFilters = filters.length > 0 || searchTerm || otherFilterConditions;

  // Apply search indexing rules
  useSearchIndexing(hasActiveFilters);

  // ... rest of component
}

​
Benefits

  1. Prevents Duplicate Content: Search engines won’t index multiple versions of the same page
  2. Maintains Shareability: URLs with filters still work for users sharing links
  3. Preserves SEO: Base pages retain their search ranking
  4. Automatic: Works without manual intervention once implemented

​
Implementation Details

  • Canonical URLs tell search engines which version is the “master”
  • Noindex prevents duplicate content while preserving link equity
  • Robots.txt provides an additional layer of protection
  • Cleanup ensures meta tags are removed when filters are cleared

​
Best Practices

  1. Always use the hook for any component that modifies URLs based on user interaction
  2. Test thoroughly to ensure meta tags are added/removed correctly
  3. Monitor search console for any remaining duplicate content issues
  4. Consider analytics when deciding which filtered pages (if any) should be indexed

​
Monitoring

Check Google Search Console and other search tools for:
  • Duplicate content warnings
  • Pages with canonical issues
  • Unexpected indexed URLs with query parameters

​
References

This approach follows industry best practices used by:
  • Major ecommerce sites (Amazon, eBay)
  • Documentation platforms (Algolia, GitLab)
  • Content management systems (WordPress, Drupal)
See Webflow’s pagination approach and ecommerce faceted navigation best practices for more details.