Ultra fast python binding for curl-impersonate via cython
Project description
curl_cffi
Python binding for curl-impersonate via cffi.
Documentation | 中文 README | Discuss on Telegram
Unlike other pure python http clients like httpx
or requests
, curl_cffi
can
impersonate browsers' TLS signatures or JA3 fingerprints. If you are blocked by some
website for no obvious reason, you can give this package a try.
Features
- Supports JA3/TLS and http2 fingerprints impersonation.
- Much faster than requests/httpx, on par with aiohttp/pycurl, see benchmarks.
- Mimics requests API, no need to learn another one.
- Pre-compiled, so you don't have to compile on your machine.
- Supports
asyncio
with proxy rotation on each request. - Supports http 2.0, which requests does not.
- Supports websocket.
library | requests | aiohttp | httpx | pycurl | curl_cffi |
---|---|---|---|---|---|
http2 | ❌ | ❌ | ✅ | ✅ | ✅ |
sync | ✅ | ❌ | ✅ | ✅ | ✅ |
async | ❌ | ✅ | ✅ | ❌ | ✅ |
websocket | ❌ | ✅ | ❌ | ❌ | ✅ |
fingerprints | ❌ | ❌ | ❌ | ❌ | ✅ |
speed | 🐇 | 🐇🐇 | 🐇 | 🐇🐇 | 🐇🐇 |
Install
pip install curl_cffi --upgrade
This should work on Linux(x86_64/aarch64), macOS(Intel/Apple Silicon) and Windows(amd64).
If it does not work on you platform, you may need to compile and install curl-impersonate
first and set some environment variables like LD_LIBRARY_PATH
.
To install beta releases:
pip install curl_cffi --upgrade --pre
To install unstable version from GitHub:
git clone https://github.com/yifeikong/curl_cffi/
cd curl_cffi
make preprocess
pip install .
Usage
Use the latest impersonate versions, do NOT copy chrome110
here without changing.
requests-like
from cycurl import requests
# Notice the impersonate parameter
r = requests.get("https://tls.browserleaks.com/json", impersonate="chrome110")
print(r.json())
# output: {..., "ja3n_hash": "aa56c057ad164ec4fdcb7a5a283be9fc", ...}
# the js3n fingerprint should be the same as target browser
# http/socks proxies are supported
proxies = {"https": "http://localhost:3128"}
r = requests.get("https://tls.browserleaks.com/json", impersonate="chrome110", proxies=proxies)
proxies = {"https": "socks://localhost:3128"}
r = requests.get("https://tls.browserleaks.com/json", impersonate="chrome110", proxies=proxies)
Sessions
# sessions are supported
s = requests.Session()
# httpbin is a http test website
s.get("https://httpbin.org/cookies/set/foo/bar")
print(s.cookies)
# <Cookies[<Cookie foo=bar for httpbin.org />]>
r = s.get("https://httpbin.org/cookies")
print(r.json())
# {'cookies': {'foo': 'bar'}}
Supported impersonate versions, as supported by my fork of curl-impersonate:
However, only Chrome-like browsers are supported. Firefox support is tracked in #59.
- chrome99
- chrome100
- chrome101
- chrome104
- chrome107
- chrome110
- chrome116 [1]
- chrome119 [1]
- chrome120 [1]
- chrome99_android
- edge99
- edge101
- safari15_3 [2]
- safari15_5 [2]
- safari17_0 [1]
- safari17_2_ios [1]
Notes:
- Added in version
0.6.0
. - fixed in version
0.6.0
, previous http2 fingerprints were not correct.
asyncio
from cycurl.requests import AsyncSession
async with AsyncSession() as s:
r = await s.get("https://example.com")
More concurrency:
import asyncio
from cycurl.requests import AsyncSession
urls = [
"https://google.com/",
"https://facebook.com/",
"https://twitter.com/",
]
async with AsyncSession() as s:
tasks = []
for url in urls:
task = s.get(url)
tasks.append(task)
results = await asyncio.gather(*tasks)
WebSockets
from curl_cffi.requests import Session, WebSocket
def on_message(ws: WebSocket, message):
print(message)
with Session() as s:
ws = s.ws_connect(
"wss://api.gemini.com/v1/marketdata/BTCUSD",
on_message=on_message,
)
ws.run_forever()
curl-like
Alternatively, you can use the low-level curl-like API:
from cycurl import Curl, CurlOpt
from io import BytesIO
buffer = BytesIO()
c = Curl()
c.setopt(CURLOPT_URL, b'https://tls.browserleaks.com/json')
c.setopt(CURLOPT_WRITEDATA, buffer)
c.impersonate("chrome110")
c.perform()
c.close()
body = buffer.getvalue()
print(body.decode())
See the docs for more details.
scrapy
If you are using scrapy, check out these middlewares:
Acknowledgement
- Originally forked from multippt/python_curl_cffi, which is under the MIT license.
- Headers/Cookies files are copied from httpx, which is under the BSD license.
- Asyncio support is inspired by Tornado's curl http client.
- The WebSocket API is inspired by websocket_client.
[Sponsor] Bypass Cloudflare with API
Yescaptcha is a proxy service that bypasses Cloudflare and uses the API interface to obtain verified cookies (e.g. cf_clearance
). Click here to register: https://yescaptcha.com/i/stfnIO
Sponsor
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distributions
Hashes for cycurl-0.6.0b9-cp312-cp312-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 67ba744aecfebd9cf7c60ae35ba0a2a7aec46a77d4f0ad29996e62f50eb4618d |
|
MD5 | 2a8fb10945970673b98d82f939520e6c |
|
BLAKE2b-256 | 0b7b4461fd6804ffbb15dd2cb69651abcb31a1c630032173d32a3c1456814faa |
Hashes for cycurl-0.6.0b9-cp312-cp312-manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | e696037e4115c0332ee8a7eca469ae477a0d788f3e91ff25f549cb9d14de7ae7 |
|
MD5 | b0c277dfda260e52dc42f0969b9fcdfb |
|
BLAKE2b-256 | e7cf6d08aa5e336e41389cfd9702428309731a146db895beee8b5820eacd4fc7 |
Hashes for cycurl-0.6.0b9-cp312-cp312-macosx_10_9_universal2.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | a41843e3ae4793180b6152cea14d72d8e414dcf7c78718fc62a47b1dc6bc6e30 |
|
MD5 | 40c57092b8b1b29f563e1f6fe280563e |
|
BLAKE2b-256 | e6106db617bd5779f129ad43687f72889b862e101191e649ae50c90dfc12470d |
Hashes for cycurl-0.6.0b9-cp311-cp311-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | d800d9920832e25632ecba5341168dcd2b251babb6f99aeee27aa958fd740371 |
|
MD5 | 96e8623a3191d26afab88cf734e02714 |
|
BLAKE2b-256 | e5a17d31fec7657a80bc986d260c4d9ae0afbdce7b35fd1805617a1edcf9fa82 |
Hashes for cycurl-0.6.0b9-cp311-cp311-manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 87fc8ff54fffea6c2f8260efa623f000c6728cb731fec8496761a07d769d0629 |
|
MD5 | ae984324312d9322945b31da608b5b94 |
|
BLAKE2b-256 | 1be04b571e3bdc15a54836caa357b17dcff61c320183080340baf0789a52215d |
Hashes for cycurl-0.6.0b9-cp311-cp311-macosx_10_9_universal2.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | ebc1447cb218185ace8cfee1f6c36554e7dff770267dba7e79a687768e8aa23e |
|
MD5 | da8491878f2fb308c01437c45555a1a1 |
|
BLAKE2b-256 | ea38e136cc0d0785fabb98ac87f4e02b0c6f3f429cceb4d1d39c87e22065e5aa |
Hashes for cycurl-0.6.0b9-cp310-cp310-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 2b9a4e62464226ca82dba68fe3afd75fc4b8e3ed5643864fd57a28ec23aa8cc1 |
|
MD5 | fd2a18a61bd214310262e77a5280d7d4 |
|
BLAKE2b-256 | 64902c089f6c0780c41c85118083d4f4523f15c5b1575ce664a46c923c0fc8a2 |
Hashes for cycurl-0.6.0b9-cp310-cp310-manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 1cd0dc32688d52cbcf2dc3831396b6cbaae6f37b41dde01ce853ba3402e6d7e1 |
|
MD5 | d5943c9bc9e86f691110c86e434e7afb |
|
BLAKE2b-256 | 9f07cc111bc041b0bc14a07e28e3e03ae3d8825ae619d24b9ff0d6df8489c293 |
Hashes for cycurl-0.6.0b9-cp310-cp310-macosx_11_0_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 6c05cfaa5b4dc2a5f3ee084ee1e2b5e8552b70df6a431907001d43eb18d335fe |
|
MD5 | 9d2a8346d76bbab174cf2444bcd0d250 |
|
BLAKE2b-256 | 76a6028f634a1592975f604e2a27841bfae761a1b10d3410a220ae0e3bb13bd0 |
Hashes for cycurl-0.6.0b9-cp310-cp310-macosx_10_9_universal2.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | b4d81f2dfc7c9f340b0c9c100f322e4416316dfe8baaaa7bbcbf328a6bb330b4 |
|
MD5 | d111bde44b1aab79e4bf61c35e1dc930 |
|
BLAKE2b-256 | a42b8b0989d66d565780ffb9518cca59edb9233e61bc5174246fabcee235512f |
Hashes for cycurl-0.6.0b9-cp39-cp39-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | d5ce710051c00f22a191f87b0d8025001bfc4b606b15e204991095098c56219d |
|
MD5 | a6911611edddd4dc7c5368e63a08e5b9 |
|
BLAKE2b-256 | 78758d007b9aafb08ad9890fa3809ca61469617567f9520226f0852687efa556 |
Hashes for cycurl-0.6.0b9-cp39-cp39-manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 4aed3369ae6d4addc6dd6fddbf621d66093f907563a869057d9dff36fd2e0214 |
|
MD5 | 51d7fd51fafd95e016cda2452ddc2b0c |
|
BLAKE2b-256 | 40cf1728a7fb28d2b9b85638a0ff7fba1e53c9326cf642c9e78cdfc0d8dc9822 |
Hashes for cycurl-0.6.0b9-cp39-cp39-macosx_11_0_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 2a47723e7faf25094ab61fa83ebf7a28b582180f1e85e722f6ec04805ccf5edf |
|
MD5 | 31e35354b311f4f1915f59b62b6617d9 |
|
BLAKE2b-256 | 37863cbbe61854a16a8300751e94204965bbc70d6c5a734fe9242224fd36a7af |
Hashes for cycurl-0.6.0b9-cp38-cp38-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | b6c8cb5aeee98d08162836e22b0a4d1c08a3f2d77e08a19e0b46008002258288 |
|
MD5 | 1621c74701b97edeb52acf5ce07499d0 |
|
BLAKE2b-256 | c6d3648c2083db6379162039a281b093b4a30f1e4939e2f5450cd18539d26e46 |
Hashes for cycurl-0.6.0b9-cp38-cp38-manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 232bcc69ac520a23c21c11eed334f53025d333099f1317d9fde5720e7d5f6eeb |
|
MD5 | 957ee8a7a735b2502550bac39c4b1b23 |
|
BLAKE2b-256 | dd623ac7e1e7cc69fb1bd2b7e9bf546c13041067a38cc25ed941e2cd137bdc44 |
Hashes for cycurl-0.6.0b9-cp38-cp38-macosx_11_0_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 450bdb9bf5bf9d24838edb3a321f8a3fbacee782e732cb05dcf7036f1ceec2a7 |
|
MD5 | 97e1a0af696c0f60ce24d47de535d218 |
|
BLAKE2b-256 | 2cf40ec7186b57c1bf876833483025a428382aa4d33b070a8fd0546cfd0dedec |