Rate Limit Issues

Solutions for handling and preventing rate limit errors (429) when using the BuyWhere API.

Understanding Rate Limits

BuyWhere implements rate limits to ensure fair usage and system stability:

Every API response includes rate limit headers:

• X-RateLimit-Limit: Your current limit
• X-RateLimit-Remaining: Requests remaining in current window
• X-RateLimit-Reset: Unix timestamp when limit resets

Symptoms of Rate Limiting

• Receiving HTTP 429 responses with "Too Many Requests"
• Response body containing retry_after information
• Sudden failure of previously working integrations

Diagnosis Steps

1. Check Response Headers: Look for X-RateLimit-Remaining: 0 in responses
2. Examine 429 Response: The body includes retry_after value:

   {
     "error": {
       "code": "RATE_LIMIT_EXCEEDED",
       "message": "Rate limit exceeded",
       "details": { "retry_after": 32 }
     }
   }
   

3. Review Usage Patterns: Are you making bursts of requests or sustained high-frequency calls?

Solutions and Best Practices

1. Implement Exponential Backoff with Jitter

Always implement retry logic with exponential backoff:

Python Example:

import time
import random
import requests
from requests.exceptions import RequestException

def fetch_with_retry(url, headers, max_retries=5):
    for attempt in range(max_retries):
        try:
            response = requests.get(url, headers=headers)
            
            if response.status_code == 429:
                retry_after = int(response.headers.get("Retry-After", 60))
                jitter = random.uniform(0, 1)
                wait = (2 ** attempt) * retry_after + jitter
                print(f"Rate limited. Waiting {wait:.1f}s before retry...")
                time.sleep(wait)
                continue
                
            elif response.status_code >= 500:
                # Server errors - shorter backoff
                time.sleep(2 ** attempt)
                continue
                
            else:
                return response
                
        except RequestException as e:
            if attempt == max_retries - 1:
                raise
            time.sleep(2 ** attempt)
    
    raise Exception("Max retries exceeded")

JavaScript/TypeScript Example:

async function fetchWithRetry(url, options, maxRetries = 5) {
    for (let i = 0; i < maxRetries; i++) {
        try {
            const response = await fetch(url, options);
            
            if (response.status === 429) {
                const retryAfter = parseInt(response.headers.get('Retry-After') || '60');
                const backoff = Math.pow(2, i) * retryAfter + Math.random() * 1000;
                console.log(`Rate limited. Waiting ${backoff.toFixed(0)}ms before retry...`);
                await new Promise(resolve => setTimeout(resolve, backoff));
                continue;
            }
            
            if (response.status >= 500) {
                await new Promise(resolve => setTimeout(resolve, Math.pow(2, i) * 1000));
                continue;
            }
            
            return response;
        } catch (error) {
            if (i === maxRetries - 1) throw error;
            await new Promise(resolve => setTimeout(resolve, Math.pow(2, i) * 1000));
        }
    }
    throw new Error('Max retries exceeded');
}

2. Monitor Rate Limit Headers Proactively

Check headers before making requests when possible:

def make_buywhere_request(endpoint, params=None):
    # First, check if we're close to limit (optional pre-check)
    # In practice, you'd track this from previous responses
    
    response = requests.get(
        f"https://api.buywhere.ai{endpoint}",
        headers=headers,
        params=params
    )
    
    # Always check response headers for next request
    remaining = int(response.headers.get('X-RateLimit-Remaining', 0))
    if remaining < 10:  # Get proactive when low
        print(f"Warning: Only {remaining} requests remaining this minute")
    
    return response

3. Use Batch Endpoints

Reduce request count by using batch operations:

• Instead of 100 individual product detail calls, use POST /v1/products/batch
• Instead of multiple search queries, broaden your search and filter client-side
• Use POST /v1/products/bulk-lookup for SKU/UPC/URL lookups

4. Implement Request Queuing

For applications with predictable traffic patterns:

import queue
import threading
import time

class RateLimitedRequester:
    def __init__(self, max_requests_per_minute=550):  # Stay under 600 limit
        self.max_requests = max_requests_per_minute
        self.request_queue = queue.Queue()
        self.lock = threading.Lock()
        self.requests_this_minute = 0
        self.window_start = time.time()
        
    def make_request(self, func, *args, **kwargs):
        # Wait if we're at limit
        with self.lock:
            now = time.time()
            if now - self.window_start >= 60:  # Reset window
                self.requests_this_minute = 0
                self.window_start = now
                
            if self.requests_this_minute >= self.max_requests:
                sleep_time = 60 - (now - self.window_start)
                time.sleep(sleep_time)
                self.requests_this_minute = 0
                self.window_start = time.time()
                
            self.requests_this_minute += 1
        
        # Make the actual request
        return func(*args, **kwargs)

5. Cache Aggressively

BuyWhere data updates every 5-10 minutes, so cache appropriately:

Prevention Strategies

1. Start Slow: Begin with low request rates and gradually increase while monitoring
2. Use Webhooks: For real-time updates instead of polling
3. Optimize Queries: Use specific search terms rather than broad ones
4. Limit Concurrent Requests: Don't burst 50+ requests simultaneously
5. Respect Retry-After: Always honor the retry_after value in 429 responses

Related Resources

• Rate Limits Section in Developer FAQ
• Performance Tips in Developer Guide
• SDK Retry Mechanisms