Skip to content

Compatibility with curl_cffi #112

Closed as not planned
Closed as not planned

Description

@ValueRaider

I struggle to get rate-limiting with curl_cffi sessions:

from requests_ratelimiter import LimiterMixin
from curl_cffi.requests import Session
from pyrate_limiter import Duration, RequestRate, Limiter
import time

class ChromeSession(Session):
    def __init__(self, **kwargs):
        kwargs['impersonate'] = "chrome"
        super().__init__(**kwargs)
class LimiterSession(LimiterMixin, ChromeSession):
    """Session class with rate-limiting behavior."""

limiter = Limiter(RequestRate(1, 3*Duration.SECOND))
session = LimiterSession(limiter=limiter)

URLS = []
URLS.append("https://httpbin.org/get")
URLS.append("https://amazon.com")
URLS.append("https://google.com")
URLS.append("https://query2.finance.yahoo.com/v8/finance/chart/INTC")

n = 3
for url in URLS:
    print("")
    print(f"# Testing url: {url}")
    for i in range(n):
        start = time.time()
        response = session.get(url, params={"symbol": "INTC"})
        print(f"Fetch {i+1}/{n} time = {time.time()-start:.2f}")
# Testing url: https://httpbin.org/get
Fetch 1/3 time = 2.28
Fetch 2/3 time = 0.90
Fetch 3/3 time = 5.58

# Testing url: https://amazon.com
Fetch 1/3 time = 1.09
Fetch 2/3 time = 0.72
Fetch 3/3 time = 0.71

# Testing url: https://google.com
Fetch 1/3 time = 0.37
Fetch 2/3 time = 0.19
Fetch 3/3 time = 0.17

# Testing url: https://query2.finance.yahoo.com/v8/finance/chart/INTC
Fetch 1/3 time = 0.21
Fetch 2/3 time = 0.04
Fetch 3/3 time = 0.04

Activity

  1. ValueRaider commented on May 7, 2025

    @ValueRaider
    ContributorAuthor

    And possibly related: when I use curl_cffi with requests_cache (with a sqlite backend), nothing is cached. Requests still work but the DB tables are empty.

  2. JWCook commented on May 7, 2025

    @JWCook
    Owner

    Unfortunately, both requests-ratelimiter and requests-cache are only compatible with the requests library. curl-cffi is a separate HTTP client built on libcurl.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions