Mirrors/cobalt
1
0
Fork 0
mirror of https://github.com/wukko/cobalt.git synced 2025-03-05 16:38:53 +01:00
Code Issues Projects Releases Packages Wiki Activity

Merge branch 'current' into feat/threads

Browse source
This commit is contained in:
Snazzah 2024-07-28 16:46:49 -05:00 committed by GitHub
commit bb04206e01
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
44 changed files with 1143 additions and 339 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

1
.github/FUNDING.yml vendored
View file

@ -1 +0,0 @@
custom: https://boosty.to/wukko/donate

36
.github/ISSUE_TEMPLATE/bug-main-instance.md vendored Normal file
View file

@ -0,0 +1,36 @@
---
name: main instance bug report
about: report an issue with cobalt.tools or api.cobalt.tools
title: '[short description of the bug]'
labels: main instance issue
assignees: ''
---
### bug description
clear and concise description of what the issue is.
### reproduction steps
steps to reproduce the described behavior.
here's an example of what it could look like:
1. go to '...'
2. click on '....'
3. download [media type] from [service]
4. see error
### screenshots
if applicable, add screenshots or screen recordings to support your explanation.
if not, remove this section.
### links
if applicable, add links that cause the issue. more = better.
if not, remove this section.
### platform information
- OS [e.g. iOS, windows]
- browser [e.g. chrome, safari, firefox]
- version [e.g. 115]
### additional context
add any other context about the problem here if applicable.
if not, remove this section.

30
.github/ISSUE_TEMPLATE/bug-report.md vendored
View file

@ -1,32 +1,36 @@
---
name: bug report
about: report an issue with downloads or something else
title: ''
about: report a global issue with the cobalt codebase
title: '[short description of the bug]'
labels: bug
assignees: ''
---
**bug description**
a clear and concise description of what the bug is.
### bug description
clear and concise description of what the issue is.
**reproduction steps**
steps to reproduce the behavior:
### reproduction steps
steps to reproduce the described behavior.
here's an example of what it could look like:
1. go to '...'
2. click on '....'
3. download this video: **[link here]**
3. download [media type] from [service]
4. see error
**screenshots**
if applicable, add screenshots or screen recordings to help explain your problem.
### screenshots
if applicable, add screenshots or screen recordings to support your explanation.
if not, remove this section.
**links**
### links
if applicable, add links that cause the issue. more = better.
if not, remove this section.
**platform**
### platform information
- OS [e.g. iOS, windows]
- browser [e.g. chrome, safari, firefox]
- version [e.g. 115]
**additional context**
add any other context about the problem here.
### additional context
add any other context about the problem here if applicable.
if not, remove this section.

14
.github/ISSUE_TEMPLATE/feature-request.md vendored
View file

@ -1,17 +1,15 @@
---
name: feature request
about: suggest a feature for cobalt
title: ''
title: '[short feature request description]'
labels: feature request
assignees: ''
---
**describe the feature you'd like to see**
a clear and concise description of what you want to happen.
### describe the feature you'd like to see
clear and concise description of the feature you want to see in cobalt.
**describe alternatives you've considered**
a clear and concise description of any alternative solutions or features you've considered.
**additional context**
add any other context or screenshots about the feature request here.
### additional context
if applicable, add any other context or screenshots related to the feature request here.
if not, remove this section.

12
.github/ISSUE_TEMPLATE/hosting-help.md vendored Normal file
View file

@ -0,0 +1,12 @@
---
name: instance hosting help
about: ask any question regarding cobalt instance hosting
title: '[short description of the problem]'
labels: instance hosting help
assignees: ''
---
### problem description
describe what issue you're having, clearly and concisely.
support your description with screenshots/links/etc when needed.

18
.github/ISSUE_TEMPLATE/service-request.md vendored Normal file
View file

@ -0,0 +1,18 @@
---
name: service request
about: request service support in cobalt
title: 'add support for [service name]'
labels: service request
assignees: ''
---
### service name & description
provide the service name and brief description of what it is.
### link samples for the service you'd like cobalt to support
list of links that cobalt should recognize.
could be regular video link, shared video link, mobile video link, shortened link, etc.
### additional context
any additional context or screenshots should go here. if there aren't any, just remove this part.

2
.github/test.sh vendored
View file

@ -18,7 +18,7 @@ test_api() {
-X POST \
-H "Accept: application/json" \
-H "Content-Type: application/json" \
-d '{"url":"https://www.youtube.com/watch?v=jNQXAC9IVRw"}')
-d '{"url":"https://vine.co/v/huwVJIEJW50", "isAudioOnly": true}')
echo "$API_RESPONSE"
STATUS=$(echo "$API_RESPONSE" | jq -r .status)

22
.github/workflows/fast-forward.yml vendored Normal file
View file

@ -0,0 +1,22 @@
name: fast-forward
on:
issue_comment:
types: [created, edited]
jobs:
fast-forward:
# Only run if the comment contains the /fast-forward command.
if: ${{ contains(github.event.comment.body, '/fast-forward')
&& github.event.issue.pull_request }}
runs-on: ubuntu-latest
permissions:
contents: write
pull-requests: write
issues: write
steps:
- name: Fast forwarding
uses: sequoia-pgp/fast-forward@v1
with:
merge: true
comment: 'on-error'

24
.github/workflows/test.yml vendored
View file

@ -36,3 +36,27 @@ jobs:
uses: actions/checkout@v4
- name: Run test script
run: .github/test.sh api
check-services:
name: test service functionality
runs-on: ubuntu-latest
outputs:
services: ${{ steps.checkServices.outputs.service_list }}
steps:
- name: Checkout repository
uses: actions/checkout@v4
- id: checkServices
run: npm ci && echo "service_list=$(node src/util/test-ci get-services)" >> "$GITHUB_OUTPUT"
test-services:
needs: check-services
runs-on: ubuntu-latest
strategy:
fail-fast: false
matrix:
service: ${{ fromJson(needs.check-services.outputs.services) }}
name: "test service: ${{ matrix.service }}"
steps:
- name: Checkout repository
uses: actions/checkout@v4
- run: npm ci && node src/util/test-ci run-tests-for ${{ matrix.service }}

39
CONTRIBUTING.md Normal file
View file

@ -0,0 +1,39 @@
# contributing to cobalt
if you're reading this, you are probably interested in contributing to cobalt, which we are very thankful for :3
this document serves as a guide to help you make contributions that we can merge into the cobalt codebase.
## translations
currently, we are **not accepting** translations of cobalt. this is because we are making significant changes to the frontend, and the currently used localization structure is being completely reworked. if this changes, this document will be updated.
## adding features or support for services
before putting in the effort to implement a feature, it's worth considering whether it would be appropriate to add it to cobalt. the cobalt api is built to assist people **only with downloading freely accessible content**. other functionality, such as:
- downloading paid / not publicly accessible content
- downloading content protected by DRM
- scraping unrelated information & exposing it outside of file metadata
will not be reviewed or merged.
if you plan on adding a feature or support for a service, but are unsure whether it would be appropriate, it's best to open an issue and discuss it beforehand.
## git
when contributing code to cobalt, there are a few guidelines in place to ensure that the code history is readable and comprehensible.
### clean commit messages
internally, we use a format similar to [conventional commits](https://www.conventionalcommits.org/en/v1.0.0/) - the first part signifies which part of the code you are changing (the *scope*), and the second part explains the change. for inspiration on how to write appropriate commit titles, you can take a look at the [commit history](https://github.com/imputnet/cobalt/commits/).
the scope is not strictly defined, you can write whatever you find most fitting for the particular change. suppose you are changing a small part of a more significant part of the codebase. in that case, you can specify both the larger and smaller scopes in the commit message for clarity (e.g., if you were changing something in internal streams, the commit could be something like `stream/internal: fix object not being handled properly`).
if you think a change deserves further explanation, we encourage you to write a short explanation in the commit message ([example](https://github.com/imputnet/cobalt/commit/d2e5b6542f71f3809ba94d56c26f382b5cb62762)), which will save both you and us time having to enquire about the change, and you explaining the reason behind it.
if your contribution has uninformative commit titles, you may be asked to interactively rebase your branch and amend each commit to include a meaningful title.
### clean commit history
if your branch is out of date and/or has some merge conflicts with the `current` branch, you should **rebase** it instead of merging. this prevents meaningless merge commits from being included in your branch, which would then end up in the cobalt git history.
if you find a mistake or bug in your code before it's merged or reviewed, instead of making a brand new commit to fix it, it would be preferable to amend that specific commit where the mistake was first introduced. this also helps us easily revert a commit if we discover that it introduced a bug or some unwanted behavior.
- if the commit you are fixing is the latest one, you can add your files to staging and then use `git commit --amend` to apply the change.
- if the commit is somewhere deeper in your branch, you can use `git commit --fixup=HASH`, where *`HASH`* is the commit you are fixing.
- afterward, you must interactively rebase your branch with `git rebase -i current --autosquash`.
this will open up an editor, but you don't need to do anything else except save the file and exit.
- once you do either of these things, you will need to do a **force push** to your branch with `git push --force-with-lease`.

4
README.md
View file

@ -19,11 +19,13 @@ this list is not final and keeps expanding over time. if support for a service y
| bilibili.com & bilibili.tv | ✅ | ✅ | ✅ | ➖ | ➖ |
| dailymotion | ✅ | ✅ | ✅ | ✅ | ✅ |
| instagram posts & reels | ✅ | ✅ | ✅ | ➖ | ➖ |
| facebook videos | ✅ | ❌ | ❌ | ➖ | ➖ |
| loom | ✅ | ❌ | ✅ | ✅ | ➖ |
| ok video | ✅ | ❌ | ✅ | ✅ | ✅ |
| pinterest | ✅ | ✅ | ✅ | ➖ | ➖ |
| reddit | ✅ | ✅ | ✅ | ❌ | ❌ |
| rutube | ✅ | ✅ | ✅ | ✅ | ✅ |
| snapchat stories & spotlights | ✅ | ✅ | ✅ | ➖ | ➖ |
| soundcloud | ➖ | ✅ | ➖ | ✅ | ✅ |
| streamable | ✅ | ✅ | ✅ | ➖ | ➖ |
| threads posts | ✅ | ✅ | ✅ | ➖ | ➖ |
@ -46,8 +48,10 @@ this list is not final and keeps expanding over time. if support for a service y
| service | notes or features |
| :-------- | :----- |
| instagram | supports reels, photos, and videos. lets you pick what to save from multi-media posts. |
| facebook | supports public accessible videos content only. |
| pinterest | supports photos, gifs, videos and stories. |
| reddit | supports gifs and videos. |
| snapchat | supports spotlights and stories. lets you pick what to save from stories. |
| rutube | supports yappy & private links. |
| soundcloud | supports private links. |
| threads | supports photos and videos. lets you pick what to save from multi-media posts. |

6
docs/api.md
View file

@ -49,9 +49,9 @@ item type: `object`
| key | type | variables | description |
|:--------|:---------|:--------------------------------------------------------|:---------------------------------------|
| `type` | `string` | `video` | used only if `pickerType`is `various`. |
| `type` | `string` | `video / photo / gif` | used only if `pickerType` is `various` |
| `url` | `string` | direct link to a file or a link to cobalt's live render | |
| `thumb` | `string` | item thumbnail that's displayed in the picker | used only for `video` type. |
| `thumb` | `string` | item thumbnail that's displayed in the picker | used for `video` and `gif` types |
## GET: `/api/stream`
cobalt's live render (or stream) endpoint. usually, you will receive a url to this endpoint
@ -73,5 +73,5 @@ response body type: `application/json`
| `branch` | `string` | git branch |
| `name` | `string` | server name |
| `url` | `string` | server url |
| `cors` | `int` | cors status |
| `cors` | `number` | cors status |
| `startTime` | `string` | server start time |

3
docs/run-an-instance.md
View file

@ -56,8 +56,9 @@ sudo service nscd start
| `API_LISTEN_ADDRESS` | `0.0.0.0` | `127.0.0.1` | changes address from which api server is accessible. **if you are using docker, you usually don't need to configure this.** |
| `API_URL` | ➖ | `https://api.cobalt.tools/` | changes url from which api server is accessible. <br> ***REQUIRED TO RUN THE API***. |
| `API_NAME` | `unknown` | `ams-1` | api server name that is shown in `/api/serverInfo`. |
| `API_EXTERNAL_PROXY` | ➖ | `http://user:password@127.0.0.1:8080`| url of the proxy that will be passed to [`ProxyAgent`](https://undici.nodejs.org/#/docs/api/ProxyAgent) and used for all external requests. HTTP(S) only. |
| `CORS_WILDCARD` | `1` | `0` | toggles cross-origin resource sharing. <br> `0`: disabled. `1`: enabled. |
| `CORS_URL` | not used | `https://cobalt.tools/` | cross-origin resource sharing url. api will be available only from this url if `CORS_WILDCARD` is set to `0`. |
| `CORS_URL` | not used | `https://cobalt.tools` | cross-origin resource sharing url. api will be available only from this url if `CORS_WILDCARD` is set to `0`. |
| `COOKIE_PATH` | not used | `/cookies.json` | path for cookie file relative to main folder. |
| `PROCESSING_PRIORITY` | not used | `10` | changes `nice` value* for ffmpeg subprocess. available only on unix systems. |
| `FREEBIND_CIDR` | ➖ | `2001:db8::/32` | IPv6 prefix used for randomly assigning addresses to cobalt requests. only supported on linux systems. see below for more info. |

27
package-lock.json generated
View file

@ -1,12 +1,12 @@
{
"name": "cobalt",
"version": "7.14.4",
"version": "7.15",
"lockfileVersion": 3,
"requires": true,
"packages": {
"": {
"name": "cobalt",
"version": "7.14.4",
"version": "7.15",
"license": "AGPL-3.0",
"dependencies": {
"content-disposition-header": "0.6.0",
@ -24,7 +24,7 @@
"set-cookie-parser": "2.6.0",
"undici": "^5.19.1",
"url-pattern": "1.0.3",
"youtubei.js": "^9.3.0"
"youtubei.js": "^10.2.0"
},
"engines": {
"node": ">=18"
@ -73,9 +73,9 @@
}
},
"node_modules/acorn": {
"version": "8.11.3",
"resolved": "https://registry.npmjs.org/acorn/-/acorn-8.11.3.tgz",
"integrity": "sha512-Y9rRfJG5jcKOE0CLisYbojUjIrIEE7AGMzA/Sm4BslANhbS+cDMpgBdcPT91oJ7OuJ9hYJBx59RjbhxVnrF8Xg==",
"version": "8.12.1",
"resolved": "https://registry.npmjs.org/acorn/-/acorn-8.12.1.tgz",
"integrity": "sha512-tcpGyI9zbizT9JbV6oYE477V6mTlXvvi0T0G3SNIYE2apm/G5huBa1+K89VGeovbg+jycCrfhl3ADxErOuO6Jg==",
"bin": {
"acorn": "bin/acorn"
},
@ -683,9 +683,9 @@
}
},
"node_modules/jintr": {
"version": "1.1.0",
"resolved": "https://registry.npmjs.org/jintr/-/jintr-1.1.0.tgz",
"integrity": "sha512-Tu9wk3BpN2v+kb8yT6YBtue+/nbjeLFv4vvVC4PJ7oCidHKbifWhvORrAbQfxVIQZG+67am/mDagpiGSVtvrZg==",
"version": "2.0.0",
"resolved": "https://registry.npmjs.org/jintr/-/jintr-2.0.0.tgz",
"integrity": "sha512-RiVlevxttZ4eHEYB2dXKXDXluzHfRuw0DJQGsYuKCc5IvZj5/GbOakeqVX+Bar/G9kTty9xDJREcxukurkmYLA==",
"funding": [
"https://github.com/sponsors/LuanRT"
],
@ -1123,14 +1123,15 @@
}
},
"node_modules/youtubei.js": {
"version": "9.4.0",
"resolved": "https://registry.npmjs.org/youtubei.js/-/youtubei.js-9.4.0.tgz",
"integrity": "sha512-8plCOZD2WabqWSEgZU3RjzigIIeR7sF028EERJENYrC9xO/6awpLMZfeoE1gNrNEbKcA+bzbMvonqlvBdxGdKg==",
"version": "10.2.0",
"resolved": "https://registry.npmjs.org/youtubei.js/-/youtubei.js-10.2.0.tgz",
"integrity": "sha512-JLKW9AHQ1qrTwBbre1aDkH8UJFmNcc4+kOSaVou5jSY7AzfFPFJK0yvX6afnLst0UVC9wfXHrLiNx93sutVErA==",
"funding": [
"https://github.com/sponsors/LuanRT"
],
"license": "MIT",
"dependencies": {
"jintr": "^1.1.0",
"jintr": "^2.0.0",
"tslib": "^2.5.0",
"undici": "^5.19.1"
}

4
package.json
View file

@ -1,7 +1,7 @@
{
"name": "cobalt",
"description": "save what you love",
"version": "7.14.4",
"version": "7.15",
"author": "imput",
"exports": "./src/cobalt.js",
"type": "module",
@ -40,7 +40,7 @@
"set-cookie-parser": "2.6.0",
"undici": "^5.19.1",
"url-pattern": "1.0.3",
"youtubei.js": "^9.3.0"
"youtubei.js": "^10.2.0"
},
"optionalDependencies": {
"freebind": "^0.2.2"

50
src/core/api.js
View file

@ -1,5 +1,6 @@
import cors from "cors";
import rateLimit from "express-rate-limit";
import { setGlobalDispatcher, ProxyAgent } from "undici";
import { env, version } from "../modules/config.js";
@ -87,32 +88,17 @@ export function runAPI(express, app, gitCommit, gitBranch, __dirname) {
next();
})
app.use('/api/json', express.json({
verify: (req, res, buf) => {
if (String(req.header('Accept')) === "application/json") {
if (buf.length > 720) throw new Error();
JSON.parse(buf);
} else {
throw new Error();
}
}
}))
// handle express.json errors properly (https://github.com/expressjs/express/issues/4065)
app.use('/api/json', (err, req, res, next) => {
let errorText = "invalid json body";
const acceptHeader = String(req.header('Accept')) !== "application/json";
if (err || acceptHeader) {
if (acceptHeader) errorText = "invalid accept header";
app.use('/api/json', express.json({ limit: 1024 }));
app.use('/api/json', (err, _, res, next) => {
if (err) {
return res.status(400).json({
status: "error",
text: errorText
text: "invalid json body"
});
} else {
next();
}
})
next();
});
app.post('/api/json', async (req, res) => {
const request = req.body;
@ -123,6 +109,10 @@ export function runAPI(express, app, gitCommit, gitBranch, __dirname) {
res.status(status).json(body);
}
if (!acceptRegex.test(req.header('Accept'))) {
return fail('ErrorInvalidAcceptHeader');
}
if (!acceptRegex.test(req.header('Content-Type'))) {
return fail('ErrorInvalidContentType');
}
@ -196,10 +186,10 @@ export function runAPI(express, app, gitCommit, gitBranch, __dirname) {
return res.sendStatus(404);
}
streamInfo.headers = {
...streamInfo.headers,
...req.headers
};
streamInfo.headers = new Map([
...(streamInfo.headers || []),
...Object.entries(req.headers)
]);
return stream(res, { type: 'internal', ...streamInfo });
})
@ -219,6 +209,14 @@ export function runAPI(express, app, gitCommit, gitBranch, __dirname) {
randomizeCiphers();
setInterval(randomizeCiphers, 1000 * 60 * 30); // shuffle ciphers every 30 minutes
if (env.externalProxy) {
if (env.freebindCIDR) {
throw new Error('Freebind is not available when external proxy is enabled')
}
setGlobalDispatcher(new ProxyAgent(env.externalProxy))
}
app.listen(env.apiPort, env.listenAddress, () => {
console.log(`\n` +
`${Cyan("cobalt")} API ${Bright(`v.${version}-${gitCommit} (${gitBranch})`)}\n` +

15
src/core/web.js
View file

@ -26,7 +26,13 @@ export async function runWeb(express, app, gitCommit, gitBranch, __dirname) {
app.get('/onDemand', (req, res) => {
try {
if (req.query.blockId) {
if (typeof req.query.blockId !== 'string') {
return res.status(400).json({
status: "error",
text: "couldn't render this block, please try again!"
});
}
let blockId = req.query.blockId.slice(0, 3);
let blockData;
switch(blockId) {
@ -54,17 +60,12 @@ export async function runWeb(express, app, gitCommit, gitBranch, __dirname) {
})
break;
}
if (blockData?.body) {
return res.status(blockData.status).json(blockData.body);
} else {
return res.status(204).end();
}
} else {
return res.status(400).json({
status: "error",
text: "couldn't render this block, please try again!"
});
}
} catch {
return res.status(400).json({
status: "error",

3
src/localization/languages/en.json
View file

@ -159,6 +159,7 @@
"UpdateOneMillion": "1 million users and blazing speed",
"ErrorYTAgeRestrict": "this youtube video is age-restricted, so i can't see it. try another one!",
"ErrorYTLogin": "couldn't get this youtube video because it requires an account to view.\n\nthis limitation is done by google to seemingly stop scraping, affecting all 3rd party tools and even their own clients.\n\ntry again, but if issue persists, {ContactLink}.",
"ErrorYTRateLimit": "i got rate limited by youtube. try again in a few seconds, but if issue persists, {ContactLink}."
"ErrorYTRateLimit": "i got rate limited by youtube. try again in a few seconds, but if issue persists, {ContactLink}.",
"ErrorInvalidAcceptHeader": "invalid accept header"
}
}

4
src/modules/config.js
View file

@ -48,7 +48,9 @@ const
processingPriority: process.platform !== 'win32'
&& process.env.PROCESSING_PRIORITY
&& parseInt(process.env.PROCESSING_PRIORITY)
&& parseInt(process.env.PROCESSING_PRIORITY),
externalProxy: process.env.API_EXTERNAL_PROXY,
}
export const

15
src/modules/processing/match.js
View file

@ -24,8 +24,10 @@ import streamable from "./services/streamable.js";
import twitch from "./services/twitch.js";
import rutube from "./services/rutube.js";
import dailymotion from "./services/dailymotion.js";
import snapchat from "./services/snapchat.js";
import loom from "./services/loom.js";
import threads from "./services/threads.js";
import facebook from "./services/facebook.js";
let freebind;
@ -189,6 +191,12 @@ export default async function(host, patternMatch, lang, obj) {
case "dailymotion":
r = await dailymotion(patternMatch);
break;
case "snapchat":
r = await snapchat({
hostname: url.hostname,
...patternMatch
});
break;
case "loom":
r = await loom({
id: patternMatch.id
@ -199,7 +207,12 @@ export default async function(host, patternMatch, lang, obj) {
...patternMatch,
quality: obj.vQuality,
dispatcher
})
});
break;
case "facebook":
r = await facebook({
...patternMatch
});
break;
default:
return createResponse("error", {

17
src/modules/processing/matchActionDecider.js
View file

@ -24,7 +24,7 @@ export default function(r, host, userFormat, isAudioOnly, lang, isAudioMuted, di
else if (r.isGif && toGif) action = "gif";
else if (isAudioMuted) action = "muteVideo";
else if (isAudioOnly) action = "audio";
else if (r.isM3U8) action = "singleM3U8";
else if (r.isM3U8) action = "m3u8";
else action = "video";
if (action === "picker" || action === "audio") {
@ -48,13 +48,19 @@ export default function(r, host, userFormat, isAudioOnly, lang, isAudioMuted, di
params = { type: "gif" }
break;
case "singleM3U8":
params = { type: "remux" }
case "m3u8":
params = {
type: Array.isArray(r.urls) ? "render" : "remux"
}
break;
case "muteVideo":
let muteType = "mute";
if (Array.isArray(r.urls) && !r.isM3U8) {
muteType = "bridge";
}
params = {
type: Array.isArray(r.urls) ? "bridge" : "mute",
type: muteType,
u: Array.isArray(r.urls) ? r.urls[0] : r.urls,
mute: true
}
@ -68,6 +74,7 @@ export default function(r, host, userFormat, isAudioOnly, lang, isAudioMuted, di
case "instagram":
case "twitter":
case "threads":
case "snapchat":
params = { picker: r.picker };
break;
case "tiktok":
@ -125,11 +132,13 @@ export default function(r, host, userFormat, isAudioOnly, lang, isAudioMuted, di
params = { type: "bridge" };
break;
case "facebook":
case "vine":
case "instagram":
case "tumblr":
case "pinterest":
case "streamable":
case "snapchat":
case "loom":
case "threads":
responseType = "redirect";

56
src/modules/processing/services/facebook.js Normal file
View file

@ -0,0 +1,56 @@
import { genericUserAgent } from "../../config.js";
const headers = {
'User-Agent': genericUserAgent,
'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,*/*;q=0.8',
'Accept-Language': 'en-US,en;q=0.5',
'Sec-Fetch-Mode': 'navigate',
'Sec-Fetch-Site': 'none',
}
const resolveUrl = (url) => {
return fetch(url, { headers })
.then(r => {
if (r.headers.get('location')) {
return decodeURIComponent(r.headers.get('location'));
}
if (r.headers.get('link')) {
const linkMatch = r.headers.get('link').match(/<(.*?)\/>/);
return decodeURIComponent(linkMatch[1]);
}
return false;
})
.catch(() => false);
}
export default async function({ id, shareType, shortLink }) {
let url = `https://web.facebook.com/i/videos/${id}`;
if (shareType) url = `https://web.facebook.com/share/${shareType}/${id}`;
if (shortLink) url = await resolveUrl(`https://fb.watch/${shortLink}`);
const html = await fetch(url, { headers })
.then(r => r.text())
.catch(() => false);
if (!html) return { error: 'ErrorCouldntFetch' };
const urls = [];
const hd = html.match('"browser_native_hd_url":(".*?")');
const sd = html.match('"browser_native_sd_url":(".*?")');
if (hd?.[1]) urls.push(JSON.parse(hd[1]));
if (sd?.[1]) urls.push(JSON.parse(sd[1]));
if (!urls.length) {
return { error: 'ErrorEmptyDownload' };
}
const baseFilename = `facebook_${id || shortLink}`;
return {
urls: urls[0],
filename: `${baseFilename}.mp4`,
audioFilename: `${baseFilename}_audio`,
};
}

6
src/modules/processing/services/instagram.js
View file

@ -256,11 +256,11 @@ export default function(obj) {
if (!media_id && cookie) media_id = await getMediaId(id, { cookie });
// mobile api (bearer)
if (media_id && token) data = await requestMobileApi(id, { token });
if (media_id && token) data = await requestMobileApi(media_id, { token });
// mobile api (no cookie, cookie)
if (!data && media_id) data = await requestMobileApi(id);
if (!data && media_id && cookie) data = await requestMobileApi(id, { cookie });
if (media_id && !data) data = await requestMobileApi(media_id);
if (media_id && cookie && !data) data = await requestMobileApi(media_id, { cookie });
// html embed (no cookie, cookie)
if (!data) data = await requestHTML(id);

13
src/modules/processing/services/ok.js
View file

@ -20,14 +20,15 @@ export default async function(o) {
}).then(r => r.text()).catch(() => {});
if (!html) return { error: 'ErrorCouldntFetch' };
if (!html.includes(`<div data-module="OKVideo" data-options="{`)) {
let videoData = html.match(/<div data-module="OKVideo" .*? data-options="({.*?})"( .*?)>/)
?.[1]
?.replaceAll("&quot;", '"');
if (!videoData) {
return { error: 'ErrorEmptyDownload' };
}
let videoData = html.split(`<div data-module="OKVideo" data-options="`)[1]
.split('" data-')[0]
.replaceAll("&quot;", '"');
videoData = JSON.parse(JSON.parse(videoData).flashvars.metadata);
if (videoData.provider !== "UPLOADED_ODKL")
@ -44,7 +45,7 @@ export default async function(o) {
let fileMetadata = {
title: cleanString(videoData.movie.title.trim()),
author: cleanString(videoData.author.name.trim()),
author: cleanString((videoData.author?.name || videoData.compilationTitle).trim()),
}
if (bestVideo) return {

2
src/modules/processing/services/pinterest.js
View file

@ -1,6 +1,6 @@
import { genericUserAgent } from "../../config.js";
const videoRegex = /"url":"(https:\/\/v1.pinimg.com\/videos\/.*?)"/g;
const videoRegex = /"url":"(https:\/\/v1\.pinimg\.com\/videos\/.*?)"/g;
const imageRegex = /src="(https:\/\/i\.pinimg\.com\/.*\.(jpg|gif))"/g;
export default async function(o) {

24
src/modules/processing/services/rutube.js
View file

@ -10,6 +10,8 @@ async function requestJSON(url) {
} catch {}
}
const delta = (a, b) => Math.abs(a - b);
export default async function(obj) {
if (obj.yappyId) {
const yappy = await requestJSON(
@ -25,7 +27,7 @@ export default async function(obj) {
}
}
const quality = obj.quality === "max" ? "9000" : obj.quality;
const quality = Number(obj.quality) || 9000;
const requestURL = new URL(`https://rutube.ru/api/play/options/${obj.id}/?no_404=true&referer&pver=v2`);
if (obj.key) requestURL.searchParams.set('p', obj.key);
@ -45,12 +47,16 @@ export default async function(obj) {
if (!m3u8) return { error: 'ErrorCouldntFetch' };
m3u8 = HLS.parse(m3u8).variants.sort((a, b) => Number(b.bandwidth) - Number(a.bandwidth));
m3u8 = HLS.parse(m3u8).variants;
let bestQuality = m3u8[0];
if (Number(quality) < bestQuality.resolution.height) {
bestQuality = m3u8.find((i) => (Number(quality) === i.resolution.height));
}
const matchingQuality = m3u8.reduce((prev, next) => {
const diff = {
prev: delta(quality, prev.resolution.height),
next: delta(quality, next.resolution.height)
};
return diff.prev < diff.next ? prev : next;
});
const fileMetadata = {
title: cleanString(play.title.trim()),
@ -58,15 +64,15 @@ export default async function(obj) {
}
return {
urls: bestQuality.uri,
urls: matchingQuality.uri,
isM3U8: true,
filenameAttributes: {
service: "rutube",
id: obj.id,
title: fileMetadata.title,
author: fileMetadata.artist,
resolution: `${bestQuality.resolution.width}x${bestQuality.resolution.height}`,
qualityLabel: `${bestQuality.resolution.height}p`,
resolution: `${matchingQuality.resolution.width}x${matchingQuality.resolution.height}`,
qualityLabel: `${matchingQuality.resolution.height}p`,
extension: "mp4"
},
fileMetadata: fileMetadata

96
src/modules/processing/services/snapchat.js Normal file
View file

@ -0,0 +1,96 @@
import { genericUserAgent } from "../../config.js";
import { getRedirectingURL } from "../../sub/utils.js";
import { extract, normalizeURL } from "../url.js";
const SPOTLIGHT_VIDEO_REGEX = /<link data-react-helmet="true" rel="preload" href="(https:\/\/cf-st\.sc-cdn\.net\/d\/[\w.?=]+&amp;uc=\d+)" as="video"\/>/;
const NEXT_DATA_REGEX = /<script id="__NEXT_DATA__" type="application\/json">({.+})<\/script><\/body><\/html>$/;
async function getSpotlight(id) {
const html = await fetch(`https://www.snapchat.com/spotlight/${id}`, {
headers: { 'User-Agent': genericUserAgent }
}).then((r) => r.text()).catch(() => null);
if (!html) {
return { error: 'ErrorCouldntFetch' };
}
const videoURL = html.match(SPOTLIGHT_VIDEO_REGEX)?.[1];
if (videoURL) {
return {
urls: videoURL,
filename: `snapchat_${id}.mp4`,
audioFilename: `snapchat_${id}_audio`
}
}
}
async function getStory(username, storyId) {
const html = await fetch(`https://www.snapchat.com/add/${username}${storyId ? `/${storyId}` : ''}`, {
headers: { 'User-Agent': genericUserAgent }
}).then((r) => r.text()).catch(() => null);
if (!html) {
return { error: 'ErrorCouldntFetch' };
}
const nextDataString = html.match(NEXT_DATA_REGEX)?.[1];
if (nextDataString) {
const data = JSON.parse(nextDataString);
const storyIdParam = data.query.profileParams[1];
if (storyIdParam && data.props.pageProps.story) {
const story = data.props.pageProps.story.snapList.find((snap) => snap.snapId.value === storyIdParam);
if (story) {
if (story.snapMediaType === 0) {
return {
urls: story.snapUrls.mediaUrl,
isPhoto: true
}
}
return {
urls: story.snapUrls.mediaUrl,
filename: `snapchat_${storyId}.mp4`,
audioFilename: `snapchat_${storyId}_audio`
}
}
}
const defaultStory = data.props.pageProps.curatedHighlights[0];
if (defaultStory) {
return {
picker: defaultStory.snapList.map((snap) => ({
type: snap.snapMediaType === 0 ? 'photo' : 'video',
url: snap.snapUrls.mediaUrl,
thumb: snap.snapUrls.mediaPreviewUrl.value
}))
}
}
}
}
export default async function (obj) {
let params = obj;
if (obj.hostname === 't.snapchat.com' && obj.shortLink) {
const link = await getRedirectingURL(`https://t.snapchat.com/${obj.shortLink}`);
if (!link?.startsWith('https://www.snapchat.com/')) {
return { error: 'ErrorCouldntFetch' };
}
const extractResult = extract(normalizeURL(link));
if (extractResult?.host !== 'snapchat') {
return { error: 'ErrorCouldntFetch' };
}
params = extractResult.patternMatch;
}
if (params.spotlightId) {
const result = await getSpotlight(params.spotlightId);
if (result) return result;
} else if (params.username) {
const result = await getStory(params.username, params.storyId);
if (result) return result;
}
return { error: 'ErrorCouldntFetch' };
}

4
src/modules/processing/services/soundcloud.js
View file

@ -18,7 +18,9 @@ async function findClientID() {
for (let script of scripts) {
let url = script[1];
if (url && !url.startsWith('https://a-v2.sndcdn.com')) return;
if (!url?.startsWith('https://a-v2.sndcdn.com/')) {
return;
}
let scrf = await fetch(url).then(r => r.text()).catch(() => {});
let id = scrf.match(/\("client_id=[A-Za-z0-9]{32}"\)/);

27
src/modules/processing/services/twitter.js
View file

@ -147,7 +147,6 @@ export default async function({ id, index, toGif, dispatcher }) {
}
let media = (repostedTweet?.media || baseTweet?.extended_entities?.media);
media = media?.filter(m => m.video_info?.variants?.length);
// check if there's a video at given index (/video/<index>)
if (index >= 0 && index < media?.length) {
@ -159,18 +158,38 @@ export default async function({ id, index, toGif, dispatcher }) {
case 0:
return { error: 'ErrorNoVideosInTweet' };
case 1:
if (media[0].type === "photo") {
return {
type: "normal",
isPhoto: true,
urls: `${media[0].media_url_https}?name=4096x4096`
}
}
return {
type: needsFixing(media[0]) ? "remux" : "normal",
urls: bestQuality(media[0].video_info.variants),
filename: `twitter_${id}.mp4`,
audioFilename: `twitter_${id}_audio`,
isGif: media[0].type === "animated_gif"
};
}
default:
const picker = media.map((content, i) => {
if (content.type === "photo") {
let url = `${content.media_url_https}?name=4096x4096`;
return {
type: "photo",
url,
thumb: url,
}
}
let url = bestQuality(content.video_info.variants);
const shouldRenderGif = content.type === 'animated_gif' && toGif;
let type = "video";
if (shouldRenderGif) type = "gif";
if (needsFixing(content) || shouldRenderGif) {
url = createStream({
service: 'twitter',
@ -181,9 +200,9 @@ export default async function({ id, index, toGif, dispatcher }) {
}
return {
type: 'video',
type,
url,
thumb: content.media_url_https,
thumb: content.media_url_https
}
});
return { picker };

230
src/modules/processing/services/vimeo.js
View file

@ -1,109 +1,169 @@
import { env } from "../../config.js";
import { cleanString } from '../../sub/utils.js';
import { cleanString, merge } from '../../sub/utils.js';
import HLS from "hls-parser";
const resolutionMatch = {
"3840": "2160",
"2732": "1440",
"2560": "1440",
"2048": "1080",
"1920": "1080",
"1366": "720",
"1280": "720",
"960": "480",
"640": "360",
"426": "240"
"3840": 2160,
"2732": 1440,
"2560": 1440,
"2048": 1080,
"1920": 1080,
"1366": 720,
"1280": 720,
"960": 480,
"640": 360,
"426": 240
}
const qualityMatch = {
"2160": "4K",
"1440": "2K",
"480": "540",
"4K": "2160",
"2K": "1440",
"540": "480"
}
export default async function(obj) {
let quality = obj.quality === "max" ? "9000" : obj.quality;
if (!quality || obj.isAudioOnly) quality = "9000";
const url = new URL(`https://player.vimeo.com/video/${obj.id}/config`);
if (obj.password) {
url.searchParams.set('h', obj.password);
const requestApiInfo = (videoId, password) => {
if (password) {
videoId += `:${password}`
}
let api = await fetch(url)
return fetch(
`https://api.vimeo.com/videos/${videoId}`,
{
headers: {
Accept: 'application/vnd.vimeo.*+json; version=3.4.2',
'User-Agent': 'Vimeo/10.19.0 (com.vimeo; build:101900.57.0; iOS 17.5.1) Alamofire/5.9.0 VimeoNetworking/5.0.0',
Authorization: 'Basic MTMxNzViY2Y0NDE0YTQ5YzhjZTc0YmU0NjVjNDQxYzNkYWVjOWRlOTpHKzRvMmgzVUh4UkxjdU5FRW80cDNDbDhDWGR5dVJLNUJZZ055dHBHTTB4V1VzaG41bEx1a2hiN0NWYWNUcldSSW53dzRUdFRYZlJEZmFoTTArOTBUZkJHS3R4V2llYU04Qnl1bERSWWxUdXRidjNqR2J4SHFpVmtFSUcyRktuQw==',
'Accept-Language': 'en'
}
}
)
.then(a => a.json())
.catch(() => {});
}
const compareQuality = (rendition, requestedQuality) => {
const quality = parseInt(rendition);
return Math.abs(quality - requestedQuality);
}
const getDirectLink = (data, quality) => {
if (!data.files) return;
const match = data.files
.filter(f => f.rendition?.endsWith('p'))
.reduce((prev, next) => {
const delta = {
prev: compareQuality(prev.rendition, quality),
next: compareQuality(next.rendition, quality)
};
return delta.prev < delta.next ? prev : next;
});
if (!match) return;
return {
urls: match.link,
filenameAttributes: {
resolution: `${match.width}x${match.height}`,
qualityLabel: match.rendition,
extension: "mp4"
}
}
}
const getHLS = async (configURL, obj) => {
if (!configURL) return;
const api = await fetch(configURL)
.then(r => r.json())
.catch(() => {});
if (!api) return { error: 'ErrorCouldntFetch' };
let downloadType = "dash";
if (!obj.isAudioOnly && JSON.stringify(api).includes('"progressive":[{'))
downloadType = "progressive";
let fileMetadata = {
title: cleanString(api.video.title.trim()),
artist: cleanString(api.video.owner.name.trim()),
}
if (downloadType !== "dash") {
if (qualityMatch[quality]) quality = qualityMatch[quality];
let all = api.request.files.progressive.sort((a, b) => Number(b.width) - Number(a.width));
let best = all[0];
let bestQuality = all[0].quality.split('p')[0];
if (qualityMatch[bestQuality]) {
bestQuality = qualityMatch[bestQuality]
}
if (Number(quality) < Number(bestQuality)) {
best = all.find(i => i.quality.split('p')[0] === quality);
}
if (!best) return { error: 'ErrorEmptyDownload' };
return {
urls: best.url,
audioFilename: `vimeo_${obj.id}_audio`,
filename: `vimeo_${obj.id}_${best.width}x${best.height}.mp4`
}
}
if (api.video.duration > env.durationLimit)
if (api.video?.duration > env.durationLimit) {
return { error: ['ErrorLengthLimit', env.durationLimit / 60] };
let masterJSONURL = api.request.files.dash.cdns.akfire_interconnect_quic.url;
let masterJSON = await fetch(masterJSONURL).then(r => r.json()).catch(() => {});
if (!masterJSON) return { error: 'ErrorCouldntFetch' };
if (!masterJSON.video) return { error: 'ErrorEmptyDownload' };
let masterJSON_Video = masterJSON.video
.sort((a, b) => Number(b.width) - Number(a.width))
.filter(a => ["dash", "mp42"].includes(a.format));
let bestVideo = masterJSON_Video[0];
if (Number(quality) < Number(resolutionMatch[bestVideo.width])) {
bestVideo = masterJSON_Video.find(i => resolutionMatch[i.width] === quality)
}
let masterM3U8 = `${masterJSONURL.split("/sep/")[0]}/sep/video/${bestVideo.id}/master.m3u8`;
const fallbackResolution = bestVideo.height > bestVideo.width ? bestVideo.width : bestVideo.height;
const urlMasterHLS = api.request?.files?.hls?.cdns?.akfire_interconnect_quic?.url;
if (!urlMasterHLS) return { error: 'ErrorCouldntFetch' }
const masterHLS = await fetch(urlMasterHLS)
.then(r => r.text())
.catch(() => {});
if (!masterHLS) return { error: 'ErrorCouldntFetch' };
const variants = HLS.parse(masterHLS)?.variants?.sort(
(a, b) => Number(b.bandwidth) - Number(a.bandwidth)
);
if (!variants || variants.length === 0) return { error: 'ErrorEmptyDownload' };
let bestQuality;
if (obj.quality < resolutionMatch[variants[0]?.resolution?.width]) {
bestQuality = variants.find(v =>
(obj.quality === resolutionMatch[v.resolution.width])
);
}
if (!bestQuality) bestQuality = variants[0];
const expandLink = (path) => {
return new URL(path, urlMasterHLS).toString();
};
let urls = expandLink(bestQuality.uri);
const audioPath = bestQuality?.audio[0]?.uri;
if (audioPath) {
urls = [
urls,
expandLink(audioPath)
]
} else if (obj.isAudioOnly) {
return { error: 'ErrorEmptyDownload' };
}
return {
urls: masterM3U8,
urls,
isM3U8: true,
fileMetadata: fileMetadata,
filenameAttributes: {
resolution: `${bestQuality.resolution.width}x${bestQuality.resolution.height}`,
qualityLabel: `${resolutionMatch[bestQuality.resolution.width]}p`,
extension: "mp4"
}
}
}
export default async function(obj) {
let quality = obj.quality === "max" ? 9000 : Number(obj.quality);
if (quality < 240) quality = 240;
if (!quality || obj.isAudioOnly) quality = 9000;
const info = await requestApiInfo(obj.id, obj.password);
let response;
if (obj.isAudioOnly) {
response = await getHLS(info.config_url, { ...obj, quality });
}
if (!response) response = getDirectLink(info, quality);
if (!response) response = { error: 'ErrorEmptyDownload' };
if (response.error) {
return response;
}
const fileMetadata = {
title: cleanString(info.name),
artist: cleanString(info.user.name),
};
return merge(
{
fileMetadata,
filenameAttributes: {
service: "vimeo",
id: obj.id,
title: fileMetadata.title,
author: fileMetadata.artist,
resolution: `${bestVideo.width}x${bestVideo.height}`,
qualityLabel: `${resolutionMatch[bestVideo.width] || fallbackResolution}p`,
extension: "mp4"
}
}
},
response
);
}

145
src/modules/processing/services/youtube.js
View file

@ -1,25 +1,27 @@
import { Innertube, Session } from 'youtubei.js';
import { env } from '../../config.js';
import { cleanString } from '../../sub/utils.js';
import { fetch } from 'undici'
import { getCookie, updateCookieValues } from '../cookie/manager.js'
import { fetch } from "undici";
import { Innertube, Session } from "youtubei.js";
import { env } from "../../config.js";
import { cleanString } from "../../sub/utils.js";
import { getCookie, updateCookieValues } from "../cookie/manager.js";
const ytBase = Innertube.create().catch(e => e);
const codecMatch = {
h264: {
codec: "avc1",
aCodec: "mp4a",
videoCodec: "avc1",
audioCodec: "mp4a",
container: "mp4"
},
av1: {
codec: "av01",
aCodec: "mp4a",
videoCodec: "av01",
audioCodec: "mp4a",
container: "mp4"
},
vp9: {
codec: "vp9",
aCodec: "opus",
videoCodec: "vp9",
audioCodec: "opus",
container: "webm"
}
}
@ -28,20 +30,21 @@ const transformSessionData = (cookie) => {
if (!cookie)
return;
const values = cookie.values();
const REQUIRED_VALUES = [
'access_token', 'refresh_token',
'client_id', 'client_secret',
'expires'
];
const values = { ...cookie.values() };
const REQUIRED_VALUES = [ 'access_token', 'refresh_token' ];
if (REQUIRED_VALUES.some(x => typeof values[x] !== 'string')) {
return;
}
return {
...values,
expires: new Date(values.expires),
};
if (values.expires) {
values.expiry_date = values.expires;
delete values.expires;
} else if (!values.expiry_date) {
return;
}
return values;
}
const cloneInnertube = async (customFetch) => {
@ -70,14 +73,19 @@ const cloneInnertube = async (customFetch) => {
}
if (session.logged_in) {
await session.oauth.refreshIfRequired();
const oldExpiry = new Date(cookie.values().expires);
const newExpiry = session.oauth.credentials.expires;
if (session.oauth.shouldRefreshToken()) {
await session.oauth.refreshAccessToken();
}
const cookieValues = cookie.values();
const oldExpiry = new Date(cookieValues.expiry_date);
const newExpiry = new Date(session.oauth.oauth2_tokens.expiry_date);
if (oldExpiry.getTime() !== newExpiry.getTime()) {
updateCookieValues(cookie, {
...session.oauth.credentials,
expires: session.oauth.credentials.expires.toISOString()
...session.oauth.client_id,
...session.oauth.oauth2_tokens,
expiry_date: newExpiry.toISOString()
});
}
}
@ -88,11 +96,16 @@ const cloneInnertube = async (customFetch) => {
export default async function(o) {
const yt = await cloneInnertube(
(input, init) => fetch(input, { ...init, dispatcher: o.dispatcher })
(input, init) => fetch(input, {
...init,
dispatcher: o.dispatcher
})
);
let info, isDubbed, format = o.format || "h264";
let quality = o.quality === "max" ? "9000" : o.quality; // 9000(p) - max quality
const quality = o.quality === "max" ? "9000" : o.quality;
let info, isDubbed,
format = o.format || "h264";
function qual(i) {
if (!i.quality_label) {
@ -115,6 +128,7 @@ export default async function(o) {
if (!info) return { error: 'ErrorCantConnectToServiceAPI' };
const playability = info.playability_status;
const basicInfo = info.basic_info;
if (playability.status === 'LOGIN_REQUIRED') {
if (playability.reason.endsWith('bot')) {
@ -129,11 +143,11 @@ export default async function(o) {
}
if (playability.status !== 'OK') return { error: 'ErrorYTUnavailable' };
if (info.basic_info.is_live) return { error: 'ErrorLiveVideo' };
if (basicInfo.is_live) return { error: 'ErrorLiveVideo' };
// return a critical error if returned video is "Video Not Available"
// or a similar stub by youtube
if (info.basic_info.id !== o.id) {
if (basicInfo.id !== o.id) {
return {
error: 'ErrorCantConnectToServiceAPI',
critical: true
@ -142,11 +156,16 @@ export default async function(o) {
let bestQuality, hasAudio;
const filterByCodec = (formats) => formats.filter(e =>
e.mime_type.includes(codecMatch[format].codec) || e.mime_type.includes(codecMatch[format].aCodec)
).sort((a, b) => Number(b.bitrate) - Number(a.bitrate));
const filterByCodec = (formats) =>
formats
.filter(e =>
e.mime_type.includes(codecMatch[format].videoCodec)
|| e.mime_type.includes(codecMatch[format].audioCodec)
)
.sort((a, b) => Number(b.bitrate) - Number(a.bitrate));
let adaptive_formats = filterByCodec(info.streaming_data.adaptive_formats);
if (adaptive_formats.length === 0 && format === "vp9") {
format = "h264"
adaptive_formats = filterByCodec(info.streaming_data.adaptive_formats)
@ -156,27 +175,43 @@ export default async function(o) {
hasAudio = adaptive_formats.find(i => i.has_audio && i.content_length);
if (bestQuality) bestQuality = qual(bestQuality);
if (!bestQuality && !o.isAudioOnly || !hasAudio) return { error: 'ErrorYTTryOtherCodec' };
if (info.basic_info.duration > env.durationLimit) return { error: ['ErrorLengthLimit', env.durationLimit / 60] };
let checkBestAudio = (i) => (i.has_audio && !i.has_video),
audio = adaptive_formats.find(i => checkBestAudio(i) && !i.is_dubbed);
if ((!bestQuality && !o.isAudioOnly) || !hasAudio)
return { error: 'ErrorYTTryOtherCodec' };
if (basicInfo.duration > env.durationLimit)
return { error: ['ErrorLengthLimit', env.durationLimit / 60] };
const checkBestAudio = (i) => (i.has_audio && !i.has_video);
let audio = adaptive_formats.find(i =>
checkBestAudio(i) && i.is_original
);
if (o.dubLang) {
let dubbedAudio = adaptive_formats.find(i =>
checkBestAudio(i) && i.language === o.dubLang && i.audio_track && !i.audio_track.audio_is_default
);
checkBestAudio(i)
&& i.language === o.dubLang
&& i.audio_track
)
if (dubbedAudio) {
audio = dubbedAudio;
isDubbed = true
isDubbed = true;
}
}
if (!audio) {
audio = adaptive_formats.find(i => checkBestAudio(i));
}
let fileMetadata = {
title: cleanString(info.basic_info.title.trim()),
artist: cleanString(info.basic_info.author.replace("- Topic", "").trim()),
title: cleanString(basicInfo.title.trim()),
artist: cleanString(basicInfo.author.replace("- Topic", "").trim()),
}
if (info.basic_info.short_description && info.basic_info.short_description.startsWith("Provided to YouTube by")) {
let descItems = info.basic_info.short_description.split("\n\n");
if (basicInfo?.short_description?.startsWith("Provided to YouTube by")) {
let descItems = basicInfo.short_description.split("\n\n");
fileMetadata.album = descItems[2];
fileMetadata.copyright = descItems[3];
if (descItems[4].startsWith("Released on:")) {
@ -192,19 +227,23 @@ export default async function(o) {
youtubeDubName: isDubbed ? o.dubLang : false
}
if (hasAudio && o.isAudioOnly) return {
if (audio && o.isAudioOnly) return {
type: "render",
isAudioOnly: true,
urls: audio.decipher(yt.session.player),
filenameAttributes: filenameAttributes,
fileMetadata: fileMetadata,
bestAudio: format === "h264" ? 'm4a' : 'opus'
bestAudio: format === "h264" ? "m4a" : "opus"
}
const matchingQuality = Number(quality) > Number(bestQuality) ? bestQuality : quality,
checkSingle = i => qual(i) === matchingQuality && i.mime_type.includes(codecMatch[format].codec),
checkRender = i => qual(i) === matchingQuality && i.has_video && !i.has_audio;
checkSingle = i =>
qual(i) === matchingQuality && i.mime_type.includes(codecMatch[format].videoCodec),
checkRender = i =>
qual(i) === matchingQuality && i.has_video && !i.has_audio;
let match, type, urls;
if (!o.isAudioOnly && !o.isAudioMuted && format === 'h264') {
match = info.streaming_data.formats.find(checkSingle);
type = "bridge";
@ -212,10 +251,14 @@ export default async function(o) {
}
const video = adaptive_formats.find(checkRender);
if (!match && video) {
if (!match && video && audio) {
match = video;
type = "render";
urls = [video.decipher(yt.session.player), audio.decipher(yt.session.player)];
urls = [
video.decipher(yt.session.player),
audio.decipher(yt.session.player)
]
}
if (match) {

20
src/modules/processing/servicesConfig.json
View file

@ -33,6 +33,7 @@
"vk": {
"alias": "vk video & clips",
"patterns": ["video:userId_:videoId", "clip:userId_:videoId", "clips:duplicate?z=clip:userId_:videoId"],
"subdomains": ["m"],
"enabled": true
},
"ok": {
@ -113,6 +114,12 @@
"patterns": ["video/:id"],
"enabled": true
},
"snapchat": {
"alias": "snapchat stories & spotlights",
"subdomains": ["t", "story"],
"patterns": [":shortLink", "spotlight/:spotlightId", "add/:username/:storyId", "u/:username/:storyId", "add/:username", "u/:username"],
"enabled": true
},
"loom": {
"alias": "loom videos",
"patterns": ["share/:id"],
@ -123,6 +130,19 @@
"tld": "net",
"patterns": [":user/post/:id"],
"enabled": true
},
"facebook": {
"alias": "facebook videos",
"altDomains": ["fb.watch"],
"subdomains": ["web"],
"patterns": [
"_shortLink/:shortLink",
":username/videos/:caption/:id",
":username/videos/:id",
"reel/:id",
"share/:shareType/:id"
],
"enabled": true
}
}
}

12
src/modules/processing/servicesPatternTesters.js
View file

@ -30,6 +30,11 @@ export const testers = {
(patternMatch.author?.length <= 255 && patternMatch.song?.length <= 255)
|| patternMatch.shortLink?.length <= 32,
"snapchat": (patternMatch) =>
(patternMatch.username?.length <= 32 && (!patternMatch.storyId || patternMatch.storyId?.length <= 255))
|| patternMatch.spotlightId?.length <= 255
|| patternMatch.shortLink?.length <= 16,
"streamable": (patternMatch) =>
patternMatch.id?.length === 6,
@ -61,4 +66,11 @@ export const testers = {
"youtube": (patternMatch) =>
patternMatch.id?.length <= 11,
"facebook": (patternMatch) =>
patternMatch.shortLink?.length <= 11
|| patternMatch.username?.length <= 30
|| patternMatch.caption?.length <= 255
|| patternMatch.id?.length <= 20 && !patternMatch.shareType
|| patternMatch.id?.length <= 20 && patternMatch.shareType?.length === 1,
}

10
src/modules/processing/url.js
View file

@ -64,7 +64,17 @@ function aliasURL(url) {
if (url.hostname === 'dai.ly' && parts.length === 2) {
url = new URL(`https://dailymotion.com/video/${parts[1]}`)
}
case "facebook":
case "fb":
if (url.searchParams.get('v')) {
url = new URL(`https://web.facebook.com/user/videos/${url.searchParams.get('v')}`)
}
if (url.hostname === 'fb.watch') {
url = new URL(`https://web.facebook.com/_shortLink/${parts[1]}`)
}
break;
case "ddinstagram":
if (services.instagram.altDomains.includes(host.domain) && [null, 'd', 'g'].includes(host.subdomain)) {
url.hostname = 'instagram.com';

4
src/modules/stream/internal-hls.js
View file

@ -23,6 +23,10 @@ function transformObject(streamInfo, hlsObject) {
hlsObject.uri = createInternalStream(fullUrl.toString(), streamInfo);
if (hlsObject.map) {
hlsObject.map = transformObject(streamInfo, hlsObject.map);
}
return hlsObject;
}

60
src/modules/stream/internal.js
View file

@ -1,7 +1,6 @@
import { request } from 'undici';
import { Readable } from 'node:stream';
import { assert } from 'console';
import { getHeaders, pipe } from './shared.js';
import { closeRequest, getHeaders, pipe } from './shared.js';
import { handleHlsPlaylist, isHlsRequest } from './internal-hls.js';
const CHUNK_SIZE = BigInt(8e6); // 8 MB
@ -27,7 +26,7 @@ async function* readChunks(streamInfo, size) {
const received = BigInt(chunk.headers['content-length']);
if (received < expected / 2n) {
streamInfo.controller.abort();
closeRequest(streamInfo.controller);
}
for await (const data of chunk.body) {
@ -38,71 +37,86 @@ async function* readChunks(streamInfo, size) {
}
}
function chunkedStream(streamInfo, size) {
assert(streamInfo.controller instanceof AbortController);
const stream = Readable.from(readChunks(streamInfo, size));
return stream;
}
async function handleYoutubeStream(streamInfo, res) {
const { signal } = streamInfo.controller;
const cleanup = () => (res.end(), closeRequest(streamInfo.controller));
try {
const req = await fetch(streamInfo.url, {
headers: getHeaders('youtube'),
method: 'HEAD',
dispatcher: streamInfo.dispatcher,
signal: streamInfo.controller.signal
signal
});
streamInfo.url = req.url;
const size = BigInt(req.headers.get('content-length'));
if (req.status !== 200 || !size) {
return res.end();
return cleanup();
}
const stream = chunkedStream(streamInfo, size);
const generator = readChunks(streamInfo, size);
const abortGenerator = () => {
generator.return();
signal.removeEventListener('abort', abortGenerator);
}
signal.addEventListener('abort', abortGenerator);
const stream = Readable.from(generator);
for (const headerName of ['content-type', 'content-length']) {
const headerValue = req.headers.get(headerName);
if (headerValue) res.setHeader(headerName, headerValue);
}
pipe(stream, res, () => res.end());
pipe(stream, res, cleanup);
} catch {
res.end();
cleanup();
}
}
export async function internalStream(streamInfo, res) {
if (streamInfo.service === 'youtube') {
return handleYoutubeStream(streamInfo, res);
}
async function handleGenericStream(streamInfo, res) {
const { signal } = streamInfo.controller;
const cleanup = () => res.end();
try {
const req = await request(streamInfo.url, {
headers: {
...streamInfo.headers,
...Object.fromEntries(streamInfo.headers),
host: undefined
},
dispatcher: streamInfo.dispatcher,
signal: streamInfo.controller.signal,
signal,
maxRedirections: 16
});
res.status(req.statusCode);
req.body.on('error', () => {});
for (const [ name, value ] of Object.entries(req.headers))
res.setHeader(name, value)
if (req.statusCode < 200 || req.statusCode > 299)
return res.end();
return cleanup();
if (isHlsRequest(req)) {
await handleHlsPlaylist(streamInfo, req, res);
} else {
pipe(req.body, res, () => res.end());
pipe(req.body, res, cleanup);
}
} catch {
streamInfo.controller.abort();
closeRequest(streamInfo.controller);
cleanup();
}
}
export function internalStream(streamInfo, res) {
if (streamInfo.service === 'youtube') {
return handleYoutubeStream(streamInfo, res);
}
return handleGenericStream(streamInfo, res);
}

28
src/modules/stream/manage.js
View file

@ -1,10 +1,12 @@
import NodeCache from "node-cache";
import { randomBytes } from "crypto";
import { nanoid } from "nanoid";
import { setMaxListeners } from "node:events";
import { decryptStream, encryptStream, generateHmac } from "../sub/crypto.js";
import { env } from "../config.js";
import { strict as assert } from "assert";
import { closeRequest } from "./shared.js";
// optional dependency
const freebind = env.freebindCIDR && await import('freebind').catch(() => {});
@ -78,16 +80,36 @@ export function createInternalStream(url, obj = {}) {
}
const streamID = nanoid();
let controller = obj.controller;
if (!controller) {
controller = new AbortController();
setMaxListeners(Infinity, controller.signal);
}
let headers;
if (obj.headers) {
headers = new Map(Object.entries(obj.headers));
}
internalStreamCache[streamID] = {
url,
service: obj.service,
headers: obj.headers,
controller: new AbortController(),
headers,
controller,
dispatcher
};
let streamLink = new URL('/api/istream', `http://127.0.0.1:${env.apiPort}`);
streamLink.searchParams.set('id', streamID);
const cleanup = () => {
destroyInternalStream(streamLink);
controller.signal.removeEventListener('abort', cleanup);
}
controller.signal.addEventListener('abort', cleanup);
return streamLink.toString();
}
@ -100,7 +122,7 @@ export function destroyInternalStream(url) {
const id = url.searchParams.get('id');
if (internalStreamCache[id]) {
internalStreamCache[id].controller.abort();
closeRequest(internalStreamCache[id].controller);
delete internalStreamCache[id];
}
}

4
src/modules/stream/shared.js
View file

@ -16,6 +16,10 @@ const serviceHeaders = {
}
}
export function closeRequest(controller) {
try { controller.abort() } catch {}
}
export function closeResponse(res) {
if (!res.headersSent) {
res.sendStatus(500);

12
src/modules/stream/types.js
View file

@ -6,7 +6,7 @@ import { create as contentDisposition } from "content-disposition-header";
import { metadataManager } from "../sub/utils.js";
import { destroyInternalStream } from "./manage.js";
import { env, ffmpegArgs, hlsExceptions } from "../config.js";
import { getHeaders, closeResponse, pipe } from "./shared.js";
import { getHeaders, closeRequest, closeResponse, pipe } from "./shared.js";
function toRawHeaders(headers) {
return Object.entries(headers)
@ -14,10 +14,6 @@ function toRawHeaders(headers) {
.join('');
}
function closeRequest(controller) {
try { controller.abort() } catch {}
}
function killProcess(p) {
// ask the process to terminate itself gracefully
p?.kill('SIGTERM');
@ -29,7 +25,7 @@ function killProcess(p) {
}
function getCommand(args) {
if (!isNaN(env.processingPriority)) {
if (typeof env.processingPriority === 'number' && !isNaN(env.processingPriority)) {
return ['nice', ['-n', env.processingPriority.toString(), ffmpeg, ...args]]
}
return [ffmpeg, args]
@ -96,6 +92,10 @@ export function streamLiveRender(streamInfo, res) {
args = args.concat(ffmpegArgs[format]);
if (hlsExceptions.includes(streamInfo.service)) {
args.push('-bsf:a', 'aac_adtstoasc')
}
if (streamInfo.metadata) {
args = args.concat(metadataManager(streamInfo.metadata))
}

21
src/modules/sub/utils.js
View file

@ -44,3 +44,24 @@ export function cleanHTML(html) {
clean = clean.replace(/\n/g, '');
return clean
}
export function getRedirectingURL(url) {
return fetch(url, { redirect: 'manual' }).then((r) => {
if ([301, 302, 303].includes(r.status) && r.headers.has('location'))
return r.headers.get('location');
}).catch(() => null);
}
export function merge(a, b) {
for (const k of Object.keys(b)) {
if (Array.isArray(b[k])) {
a[k] = [...(a[k] ?? []), ...b[k]];
} else if (typeof b[k] === 'object') {
a[k] = merge(a[k], b[k]);
} else {
a[k] = b[k];
}
}
return a;
}

42
src/modules/test.js Normal file
View file

@ -0,0 +1,42 @@
import { normalizeRequest } from "../modules/processing/request.js";
import match from "./processing/match.js";
import { extract } from "./processing/url.js";
export async function runTest(url, params, expect) {
const normalized = normalizeRequest({ url, ...params });
if (!normalized) {
throw "invalid request";
}
const parsed = extract(normalized.url);
if (parsed === null) {
throw `invalid url: ${normalized.url}`;
}
const result = await match(
parsed.host, parsed.patternMatch, "en", normalized
);
let error = [];
if (expect.status !== result.body.status) {
const detail = `${expect.status} (expected) != ${result.body.status} (actual)`;
error.push(`status mismatch: ${detail}`);
}
if (expect.code !== result.status) {
const detail = `${expect.code} (expected) != ${result.status} (actual)`;
error.push(`status code mismatch: ${detail}`);
}
if (error.length) {
if (result.body.text) {
error.push(`error message: ${result.body.text}`);
}
throw error.join('\n');
}
if (result.body.status === 'stream') {
// TODO: stream testing
}
}

6
src/util/generate-youtube-tokens.js
View file

@ -20,9 +20,9 @@ tube.session.once(
);
tube.session.once('auth-error', (err) => bail('An error occurred:', err));
tube.session.once('auth', ({ status, credentials, ...rest }) => {
if (status !== 'SUCCESS') {
bail('something went wrong', rest);
tube.session.once('auth', ({ credentials }) => {
if (!credentials.access_token) {
bail('something went wrong');
}
console.log(

81
src/util/test-ci.js Normal file
View file

@ -0,0 +1,81 @@
import { env } from "../modules/config.js";
import { runTest } from "../modules/test.js";
import { loadLoc } from "../localization/manager.js";
import { loadJSON } from "../modules/sub/loadFromFs.js";
import { Red, Bright } from "../modules/sub/consoleText.js";
const tests = loadJSON('./src/util/tests.json');
const services = loadJSON('./src/modules/processing/servicesConfig.json');
// services that are known to frequently fail due to external
// factors (e.g. rate limiting)
const finnicky = new Set(['bilibili', 'instagram', 'youtube'])
const action = process.argv[2];
switch (action) {
case "get-services":
const fromConfig = Object.keys(services.config);
const missingTests = fromConfig.filter(
service => !tests[service] || tests[service].length === 0
);
if (missingTests.length) {
console.error('services have no tests:', missingTests);
console.log('[]');
process.exitCode = 1;
break;
}
console.log(JSON.stringify(fromConfig));
break;
case "run-tests-for":
const service = process.argv[3];
let failed = false;
if (!tests[service]) {
console.error('no such service:', service);
}
await loadLoc();
env.streamLifespan = 10000;
env.apiURL = 'http://x';
for (const test of tests[service]) {
const { name, url, params, expected } = test;
const canFail = test.canFail || finnicky.has(service);
try {
await runTest(url, params, expected);
console.log(`${service}/${name}: ok`);
} catch(e) {
failed = !canFail;
let failText = canFail ? `${Red('FAIL')} (ignored)` : Bright(Red('FAIL'));
if (canFail && process.env.GITHUB_ACTION) {
console.log(`::warning title=${service}/${name.replace(/,/g, ';')}::failed and was ignored`);
}
console.error(`${service}/${name}: ${failText}`);
const errorString = e.toString().split('\n');
let c = '┃';
errorString.forEach((line, index) => {
line = line.replace('!=', Red('!='));
if (index === errorString.length - 1) {
c = '┗';
}
console.error(` ${c}`, line);
});
}
}
process.exitCode = Number(failed);
break;
default:
console.error('invalid action:', action);
process.exitCode = 1;
}

147
src/util/tests.json
View file

@ -1,7 +1,7 @@
{
"twitter": [{
"name": "regular video",
"url": "https://twitter.com/TwitterSpaces/status/1526955853743546372?s=20",
"url": "https://twitter.com/X/status/1697304622749086011",
"params": {
"aFormat": "mp3",
"isAudioOnly": false,
@ -11,10 +11,9 @@
"code": 200,
"status": "redirect"
}
},
{
}, {
"name": "video with mobile web mediaviewer",
"url": "https://twitter.com/XSpaces/status/1526955853743546372/mediaViewer?currentTweet=1526955853743546372&currentTweetUser=XSpaces&currentTweet=1526955853743546372&currentTweetUser=XSpaces",
"url": "https://twitter.com/X/status/1697304622749086011/mediaViewer?currentTweet=1697304622749086011&currentTweetUser=X&currentTweet=1697304622749086011&currentTweetUser=X",
"params": {},
"expected": {
"code": 200,
@ -34,7 +33,7 @@
}
}, {
"name": "mixed media (image + gif)",
"url": "https://twitter.com/Twitter/status/1580661436132757506?s=20",
"url": "https://twitter.com/sky_mj26/status/1807756010712428565",
"params": {
"aFormat": "mp3",
"isAudioOnly": false,
@ -42,7 +41,7 @@
},
"expected": {
"code": 200,
"status": "redirect"
"status": "picker"
}
}, {
"name": "picker: mixed media (3 videos)",
@ -113,9 +112,10 @@
"status": "redirect"
}
}, {
"name": "age-restricted video",
"name": "age restricted video",
"url": "https://twitter.com/FckyeahCharli/status/1650987582749065220",
"params": {},
"canFail": true,
"expected": {
"code": 200,
"status": "redirect"
@ -136,6 +136,22 @@
"code": 200,
"status": "redirect"
}
}, {
"name": "post with 1 image",
"url": "https://x.com/PopCrave/status/1815960083475423235",
"params": {},
"expected": {
"code": 200,
"status": "redirect"
}
}, {
"name": "post with 4 images",
"url": "https://x.com/PopCrave/status/1816260887147114696",
"params": {},
"expected": {
"code": 200,
"status": "picker"
}
}, {
"name": "retweeted video, isAudioOnly",
"url": "https://twitter.com/hugekiwinuts/status/1618671150829309953?s=46&t=gItGzgwGQQJJaJrO6qc1Pg",
@ -650,23 +666,23 @@
}
}, {
"name": "1080p dash parcel",
"url": "https://vimeo.com/774694040",
"url": "https://vimeo.com/967252742",
"params": {
"vQuality": "1440"
},
"expected": {
"code": 200,
"status": "stream"
"status": "redirect"
}
}, {
"name": "720p dash parcel",
"url": "https://vimeo.com/774694040",
"url": "https://vimeo.com/967252742",
"params": {
"vQuality": "360"
},
"expected": {
"code": 200,
"status": "stream"
"status": "redirect"
}
}, {
"name": "private video",
@ -674,7 +690,15 @@
"params": {},
"expected": {
"code": 200,
"status": "stream"
"status": "redirect"
}
}, {
"name": "mature video",
"url": "https://vimeo.com/973212054",
"params": {},
"expected": {
"code": 200,
"status": "redirect"
}
}],
"reddit": [{
@ -947,7 +971,7 @@
}],
"streamable": [{
"name": "regular video",
"url": "https://streamable.com/03r3c2",
"url": "https://streamable.com/p9cln4",
"params": {},
"expected": {
"code": 200,
@ -963,7 +987,7 @@
}
}, {
"name": "regular video (isAudioOnly)",
"url": "https://streamable.com/03r3c2",
"url": "https://streamable.com/p9cln4",
"params": {
"isAudioOnly": true
},
@ -973,7 +997,7 @@
}
}, {
"name": "regular video (isAudioMuted)",
"url": "https://streamable.com/03r3c2",
"url": "https://streamable.com/p9cln4",
"params": {
"isAudioMuted": true
},
@ -1042,8 +1066,8 @@
"url": "https://rutube.ru/video/b521653b4f71ece57b8ff54e57ca9b82/",
"params": {},
"expected": {
"code": 200,
"status": "stream"
"code": 400,
"status": "error"
}
}, {
"name": "vertical video",
@ -1055,7 +1079,8 @@
}
}, {
"name": "yappy",
"url": "https://rutube.ru/yappy/f1771e86294748eea8ecb2ac88e55740/",
"url": "https://rutube.ru/yappy/c8c32bf7aee04412837656ea26c2b25b/",
"canFail": true,
"params": {},
"expected": {
"code": 200,
@ -1132,6 +1157,31 @@
"status": "stream"
}
}],
"snapchat": [{
"name": "spotlight",
"url": "https://www.snapchat.com/spotlight/W7_EDlXWTBiXAEEniNoMPwAAYdWxucG9pZmNqAY46j0a5AY46j0YbAAAAAQ",
"params": {},
"expected": {
"code": 200,
"status": "redirect"
}
}, {
"name": "shortlinked spotlight",
"url": "https://t.snapchat.com/4ZsiBLDi",
"params": {},
"expected": {
"code": 200,
"status": "redirect"
}
}, {
"name": "story",
"url": "https://www.snapchat.com/add/bazerkmakane",
"params": {},
"expected": {
"code": 200,
"status": "picker"
}
}],
"loom": [{
"name": "1080p video",
"url": "https://www.loom.com/share/313bf71d20ca47b2a35b6634cefdb761",
@ -1156,9 +1206,68 @@
"params": {
"isAudioOnly": true
},
"expected": {
"code": 400,
"status": "error"
}
}],
"facebook": [{
"name": "direct video with username and id",
"url": "https://web.facebook.com/100048111287134/videos/1157798148685638/",
"params": {},
"expected": {
"code": 200,
"status": "stream"
"status": "redirect"
}
}, {
"name": "direct video with id as query param",
"url": "https://web.facebook.com/watch/?v=883839773514682&ref=sharing",
"params": {},
"expected": {
"code": 200,
"status": "redirect"
}
}, {
"name": "direct video with caption",
"url": "https://web.facebook.com/wood57/videos/𝐒𝐞𝐛𝐚𝐬𝐤𝐨𝐦-𝐟𝐮𝐥𝐥/883839773514682",
"params": {},
"expected": {
"code": 200,
"status": "redirect"
}
}, {
"name": "shortlink video",
"url": "https://fb.watch/r1K6XHMfGT/",
"canFail": true,
"params": {},
"expected": {
"code": 200,
"status": "redirect"
}
}, {
"name": "reel video",
"url": "https://web.facebook.com/reel/730293269054758",
"canFail": true,
"params": {},
"expected": {
"code": 200,
"status": "redirect"
}
}, {
"name": "shared video link",
"url": "https://www.facebook.com/share/v/NEf87jbPTvFE8LsL/",
"params": {},
"expected": {
"code": 200,
"status": "redirect"
}
}, {
"name": "shared video link v2",
"url": "https://web.facebook.com/share/r/JFZfPVgLkiJQmWrr/",
"params": {},
"expected": {
"code": 200,
"status": "redirect"
}
}],
"threads": [{