Ultra fast python binding for curl-impersonate via cython
Project description
curl_cffi
Python binding for curl-impersonate via cffi.
Documentation | 中文 README | Discuss on Telegram
Unlike other pure python http clients like httpx
or requests
, curl_cffi
can
impersonate browsers' TLS signatures or JA3 fingerprints. If you are blocked by some
website for no obvious reason, you can give this package a try.
Features
- Supports JA3/TLS and http2 fingerprints impersonation.
- Much faster than requests/httpx, on par with aiohttp/pycurl, see benchmarks.
- Mimics requests API, no need to learn another one.
- Pre-compiled, so you don't have to compile on your machine.
- Supports
asyncio
with proxy rotation on each request. - Supports http 2.0, which requests does not.
- Supports websocket.
library | requests | aiohttp | httpx | pycurl | curl_cffi |
---|---|---|---|---|---|
http2 | ❌ | ❌ | ✅ | ✅ | ✅ |
sync | ✅ | ❌ | ✅ | ✅ | ✅ |
async | ❌ | ✅ | ✅ | ❌ | ✅ |
websocket | ❌ | ✅ | ❌ | ❌ | ✅ |
fingerprints | ❌ | ❌ | ❌ | ❌ | ✅ |
speed | 🐇 | 🐇🐇 | 🐇 | 🐇🐇 | 🐇🐇 |
Install
pip install curl_cffi --upgrade
This should work on Linux(x86_64/aarch64), macOS(Intel/Apple Silicon) and Windows(amd64).
If it does not work on you platform, you may need to compile and install curl-impersonate
first and set some environment variables like LD_LIBRARY_PATH
.
To install beta releases:
pip install curl_cffi --upgrade --pre
Usage
Use the latest impersonate versions, do NOT copy chrome110
here without changing.
requests-like
from cycurl import requests
# Notice the impersonate parameter
r = requests.get("https://tls.browserleaks.com/json", impersonate="chrome110")
print(r.json())
# output: {..., "ja3n_hash": "aa56c057ad164ec4fdcb7a5a283be9fc", ...}
# the js3n fingerprint should be the same as target browser
# http/socks proxies are supported
proxies = {"https": "http://localhost:3128"}
r = requests.get("https://tls.browserleaks.com/json", impersonate="chrome110", proxies=proxies)
proxies = {"https": "socks://localhost:3128"}
r = requests.get("https://tls.browserleaks.com/json", impersonate="chrome110", proxies=proxies)
Sessions
# sessions are supported
s = requests.Session()
# httpbin is a http test website
s.get("https://httpbin.org/cookies/set/foo/bar")
print(s.cookies)
# <Cookies[<Cookie foo=bar for httpbin.org />]>
r = s.get("https://httpbin.org/cookies")
print(r.json())
# {'cookies': {'foo': 'bar'}}
Supported impersonate versions, as supported by my fork of curl-impersonate:
However, only Chrome-like browsers are supported. Firefox support is tracked in #59.
- chrome99
- chrome100
- chrome101
- chrome104
- chrome107
- chrome110
- chrome116 [1]
- chrome119 [1]
- chrome120 [1]
- chrome99_android
- edge99
- edge101
- safari15_3 [2]
- safari15_5 [2]
- safari17_0 [1]
- safari17_2_ios [1]
Notes:
- Added in version
0.6.0
. - fixed in version
0.6.0
, previous http2 fingerprints were not correct.
asyncio
from cycurl.requests import AsyncSession
async with AsyncSession() as s:
r = await s.get("https://example.com")
More concurrency:
import asyncio
from cycurl.requests import AsyncSession
urls = [
"https://google.com/",
"https://facebook.com/",
"https://twitter.com/",
]
async with AsyncSession() as s:
tasks = []
for url in urls:
task = s.get(url)
tasks.append(task)
results = await asyncio.gather(*tasks)
WebSockets
from curl_cffi.requests import Session, WebSocket
def on_message(ws: WebSocket, message):
print(message)
with Session() as s:
ws = s.ws_connect(
"wss://api.gemini.com/v1/marketdata/BTCUSD",
on_message=on_message,
)
ws.run_forever()
curl-like
Alternatively, you can use the low-level curl-like API:
from cycurl import Curl, CurlOpt
from io import BytesIO
buffer = BytesIO()
c = Curl()
c.setopt(CURLOPT_URL, b'https://tls.browserleaks.com/json')
c.setopt(CURLOPT_WRITEDATA, buffer)
c.impersonate("chrome110")
c.perform()
c.close()
body = buffer.getvalue()
print(body.decode())
See the docs for more details.
scrapy
If you are using scrapy, check out these middlewares:
Acknowledgement
- Originally forked from multippt/python_curl_cffi, which is under the MIT license.
- Headers/Cookies files are copied from httpx, which is under the BSD license.
- Asyncio support is inspired by Tornado's curl http client.
- The WebSocket API is inspired by websocket_client.
[Sponsor] Bypass Cloudflare with API
Yescaptcha is a proxy service that bypasses Cloudflare and uses the API interface to obtain verified cookies (e.g. cf_clearance
). Click here to register: https://yescaptcha.com/i/stfnIO
Sponsor
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distributions
Hashes for cycurl-0.6.0b7-cp312-cp312-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 44d818b6b88694ce186627cbd2556abdc8e4e6a4ae183de1c20ec4655580257e |
|
MD5 | 02f0d50b7488b5c6e99f9b09ce5927e5 |
|
BLAKE2b-256 | 2f03472b8799088cf3b7e966cc5510c2f155c9051052fb204885aa2e3d8c6161 |
Hashes for cycurl-0.6.0b7-cp312-cp312-manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 7855b6f6048a7bc80f58afa31164b9ebe108eb4db66a6c2b59dda3c54717d53a |
|
MD5 | 35d8b619dbba599e768c6bd3a0fa0857 |
|
BLAKE2b-256 | 85deaab96aba32d0f5a471e80b23750d3e8a359f328c5ae70b0d47f88db99460 |
Hashes for cycurl-0.6.0b7-cp312-cp312-macosx_10_9_universal2.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | e6078ff2d49530c24fbe78a7c7635cbc138baabaf32a4e81779313c5da1df883 |
|
MD5 | ce9399e6d9476afe762b0f7fcbcde6ad |
|
BLAKE2b-256 | b7e78e778801b700c47ca0da87568f1b8dd1d9d06216e1a9783e4ac6fdce84c6 |
Hashes for cycurl-0.6.0b7-cp311-cp311-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | b03e6cc1521d3f6c665c8cfef5871ed778fb08b423e4a16b3d0219d21cbc314d |
|
MD5 | 4b8fb561042ea70e8eccfbc439658cb8 |
|
BLAKE2b-256 | f13d3475d58a47acafe2d838ff34dacb77255d7dd5188df4b2a1d76298cde56d |
Hashes for cycurl-0.6.0b7-cp311-cp311-manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | f4f63cdee8224629a89080e7fec2e70538d4f845f0cce021b12a613482a1518c |
|
MD5 | ee74d7edef1eb5c1f4a604bb06cd7719 |
|
BLAKE2b-256 | 43fd5f8b03cd269028c239cc63cfda5f2a98321ab51d7a3dd565d0197cc93157 |
Hashes for cycurl-0.6.0b7-cp311-cp311-macosx_10_9_universal2.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | da1c8610c11376fab539ff9604c8bf1350e375db11cf036f9e020e054ca63d4e |
|
MD5 | 0431ec68eede0939ab62e0e520edb641 |
|
BLAKE2b-256 | feb7f9d72993428427e767bda96397b2f3428bfba4f879d507415b2953b1016a |
Hashes for cycurl-0.6.0b7-cp310-cp310-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 2e0b2fe8350134abc0a023ee82256a2122089468b78721d42d61783e3df051f1 |
|
MD5 | b125f6eed9d71c21ea60a02e56308c45 |
|
BLAKE2b-256 | 71347a4630d62b98940fb50581f2198c135de3171b37f60e4579ef53b83f96d4 |
Hashes for cycurl-0.6.0b7-cp310-cp310-manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | ac52ea3076fa8b196142de8abf2e37aacf81240b94e046f439f8b89e556b2cbb |
|
MD5 | f56ade4974ec5b63768c5969f4b97b86 |
|
BLAKE2b-256 | abb0aa6200cd05bcfd3fee8f3196404af35726b7c7a48ee5c91effdd1d64c9d2 |
Hashes for cycurl-0.6.0b7-cp310-cp310-macosx_11_0_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 766ffb4e391f386103c411c9850677cebd1e5383b707439b556929799a1af7df |
|
MD5 | 80ec05d5049a7c8dedd0c5b0b0d329c0 |
|
BLAKE2b-256 | 6215b12d6f8883a99cfcad47097d703d5f7b5c20f886ca295c10149d244f4145 |
Hashes for cycurl-0.6.0b7-cp39-cp39-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 0e5fefc29503fed413e7cf5bfbb6ed44d96995ca59a107b701f303d4958f2877 |
|
MD5 | ff0200db9205bd79904919463036b207 |
|
BLAKE2b-256 | dec86b50222a1956439d3cae99cb6db3e38609d5996fef0a4d39a819673e0d8f |
Hashes for cycurl-0.6.0b7-cp39-cp39-manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | c1d51a509ef097fbe5cc0727fdf799d093f53e0a3c3e24fd9ac660daa0a75794 |
|
MD5 | 4fe89855cd873516469ebb21a4a25521 |
|
BLAKE2b-256 | 35b9147045a9d79fe032f798b05195ab356d412f6f3d5a30662626a531cfc7fb |
Hashes for cycurl-0.6.0b7-cp39-cp39-macosx_11_0_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 9222332d3c52ccc8af7893389d8f08f88feaa1bcbaa624e72ce5e041a4bf7a3d |
|
MD5 | 69cf064a82402106d8b9ee074b17d168 |
|
BLAKE2b-256 | 2e886521efd3f287a4b4708dcbf3fc3b9533dc40b97de3305401ed9d9ec87aa5 |
Hashes for cycurl-0.6.0b7-cp38-cp38-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | d790d5bfb9b395902fec58a59cc2979d4a18117787d0b87d8da8e1ff39dd557f |
|
MD5 | 4b0a43c3b9fa42a90a457f9bce1ca201 |
|
BLAKE2b-256 | 6da81fd8c7044e4b94864280d8e38b18ffb73ca4ce7875bfea2e30d072d905f9 |
Hashes for cycurl-0.6.0b7-cp38-cp38-manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 1de35bf3d9ba3ec4656fcd763384470509806a038eb64d193d4f4ae48345442c |
|
MD5 | 1027a964dda695c57315f2686a05736a |
|
BLAKE2b-256 | 89e43699f8d3ff9e1b160317a813aa21f61737734fa5fb6145fe80e9986e8cb3 |
Hashes for cycurl-0.6.0b7-cp38-cp38-macosx_11_0_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 72b07932ecd4868fe0ba39a32876d7fac284eeef7eb262fe26ba666a5554adfd |
|
MD5 | 7b0a468769aa06c1d0de6c517b7560b1 |
|
BLAKE2b-256 | 49d9b00245772d0130c9a2533e606dee42a691415b1fa8dc5e8830270f178789 |