7.12: private traffic stats with plausible and no watermark for tiktok videos by default (#392)

2024-11-06 00:10:02 +00:00 · 2024-03-16 23:47:08 +06:00 · 2024-03-16 23:47:08 +06:00 · 633fc39308
commit 633fc39308
parent 1fdc25e2ad 783fbee5d5
13 changed files with 99 additions and 157 deletions
--- a/README.md
+++ b/README.md
@ -89,3 +89,39 @@ you are allowed to host an ***unmodified*** instance of cobalt with branding, bu
 - [Fluent Emoji by Microsoft](https://github.com/microsoft/fluentui-emoji) (used in cobalt) is under [MIT](https://github.com/microsoft/fluentui-emoji/blob/main/LICENSE) license.  
 - [Noto Sans Mono](https://fonts.google.com/noto/specimen/Noto+Sans+Mono/) fonts (used in cobalt) are licensed under the [OFL](https://fonts.google.com/noto/specimen/Noto+Sans+Mono/about) license.  
 - many update banners were taken from [tenor.com](https://tenor.com/).  
+
+## acknowledgements
+### ffmpeg
+cobalt heavily relies on ffmpeg for converting and merging media files. it's an absolutely amazing piece of software offered for anyone for free, yet doesn't receive as much credit as it should.
+
+you can [support ffmpeg here](https://ffmpeg.org/donations.html)!
+
+#### ffmpeg-static
+we use [ffmpeg-static](https://github.com/eugeneware/ffmpeg-static) to get binaries for ffmpeg depending on the platform.
+
+you can support the developer via various methods listed on their github page! (linked above)
+
+### youtube.js
+cobalt relies on [youtube.js](https://github.com/LuanRT/YouTube.js) for interacting with the innertube api, it wouldn't have been possible.  
+
+you can support the developer via various methods listed on their github page! (linked above)
+
+### many others
+cobalt also depends on:
+
+- [content-disposition-header](https://www.npmjs.com/package/content-disposition-header) to simplify the provision of `content-disposition` headers.
+- [cors](https://www.npmjs.com/package/cors) to manage cross-origin resource sharing within expressjs.
+- [dotenv](https://www.npmjs.com/package/dotenv) to load environment variables from the `.env` file.
+- [esbuild](https://www.npmjs.com/package/esbuild) to minify the frontend files.
+- [express](https://www.npmjs.com/package/express) as the backbone of cobalt servers.
+- [express-rate-limit](https://www.npmjs.com/package/express-rate-limit) to rate limit api endpoints.
+- [hls-parser](https://www.npmjs.com/package/hls-parser) to parse `m3u8` playlists for certain services.
+- [ipaddr.js](https://www.npmjs.com/package/ipaddr.js) to parse ip addresses (for rate limiting).
+- [nanoid](https://www.npmjs.com/package/nanoid) to generate unique (temporary) identifiers for each requested stream.
+- [node-cache](https://www.npmjs.com/package/node-cache) to cache stream info in server ram for a limited amount of time.
+- [psl](https://www.npmjs.com/package/psl) as the domain name parser.
+- [set-cookie-parser](https://www.npmjs.com/package/set-cookie-parser) to parse cookies that cobalt receives from certain services.
+- [undici](https://www.npmjs.com/package/undici) for making http requests
+- [url-pattern](https://www.npmjs.com/package/url-pattern) to match provided links with supported patterns.
+
+...and many other packages that these packages rely on.
--- a/docs/api.md
+++ b/docs/api.md
@ -27,7 +27,6 @@ Content-Type: application/json
 | `aFormat`         | `string`  | `best / mp3 / ogg / wav / opus`    | `mp3`     |                                                                                 |
 | `filenamePattern` | `string`  | `classic / pretty / basic / nerdy` | `classic` | changes the way files are named. previews can be seen in the web app.           |
 | `isAudioOnly`     | `boolean` | `true / false`                     | `false`   |                                                                                 |
-| `isNoTTWatermark` | `boolean` | `true / false`                     | `false`   | changes whether downloaded tiktok videos have watermarks.                       |
 | `isTTFullAudio`   | `boolean` | `true / false`                     | `false`   | enables download of original sound used in a tiktok video.                      |
 | `isAudioMuted`    | `boolean` | `true / false`                     | `false`   | disables audio track in video downloads.                                        |
 | `dubLang`         | `boolean` | `true / false`                     | `false`   | backend uses Accept-Language header for youtube video audio tracks when `true`. |
--- a/docs/run-an-instance.md
+++ b/docs/run-an-instance.md
@ -63,10 +63,13 @@ sudo service nscd start
 \* the higher the nice value, the lower the priority. [read more here](https://en.wikipedia.org/wiki/Nice_(Unix)).

 ### variables for web
-| variable name   | default              | example                 | description                                                                           |
-|:--------------- |:---------------------|:------------------------|:--------------------------------------------------------------------------------------|
-| `WEB_PORT`      | `9001`               |  `9001`                 | changes port from which frontend server is accessible.                                |
-| `WEB_URL`       | ➖                   | `https://cobalt.tools/` | changes url from which frontend server is accessible. <br> ***REQUIRED TO RUN WEB***. |
-| `API_URL`       | `https://co.wuk.sh/` | `https://co.wuk.sh/`    | changes url which is used for api requests by frontend clients.                       |
-| `SHOW_SPONSORS` | `0`                  | `1`                     | toggles sponsor list in about popup. <br> `0`: disabled. `1`: enabled.                |
-| `IS_BETA`       | `0`                  | `1`                     | toggles beta tag next to cobalt logo. <br> `0`: disabled. `1`: enabled.               |
+| variable name        | default              | example                 | description                                                                           |
+|:---------------------|:---------------------|:------------------------|:--------------------------------------------------------------------------------------|
+| `WEB_PORT`           | `9001`               |  `9001`                 | changes port from which frontend server is accessible.                                |
+| `WEB_URL`            | ➖                   | `https://cobalt.tools/` | changes url from which frontend server is accessible. <br> ***REQUIRED TO RUN WEB***. |
+| `API_URL`            | `https://co.wuk.sh/` | `https://co.wuk.sh/`    | changes url which is used for api requests by frontend clients.                       |
+| `SHOW_SPONSORS`      | `0`                  | `1`                     | toggles sponsor list in about popup. <br> `0`: disabled. `1`: enabled.                |
+| `IS_BETA`            | `0`                  | `1`                     | toggles beta tag next to cobalt logo. <br> `0`: disabled. `1`: enabled.               |
+| `PLAUSIBLE_HOSTNAME` | ➖                   | `plausible.io`*         | enables plausible analytics with provided hostname as receiver backend.               |
+
+\* don't use plausible.io as receiver backend unless you paid for their cloud service. use your own domain when hosting community edition of plausible. refer to their [docs](https://plausible.io/docs) when needed.
--- a/package.json
+++ b/package.json
@ -1,7 +1,7 @@
 {
    "name": "cobalt",
    "description": "save what you love",
-    "version": "7.11.2",
+    "version": "7.12",
    "author": "wukko",
    "exports": "./src/cobalt.js",
    "type": "module",
@ -25,7 +25,6 @@
    },
    "homepage": "https://github.com/wukko/cobalt#readme",
    "dependencies": {
-        "abort-controller": "3.0.0",
        "content-disposition-header": "0.6.0",
        "cors": "^2.8.5",
        "dotenv": "^16.0.1",
--- a/src/front/cobalt.js
+++ b/src/front/cobalt.js
@ -24,13 +24,13 @@ const checkboxes = [
    "alwaysVisibleButton",
    "disableChangelog",
    "downloadPopup",
-    "disableTikTokWatermark",
    "fullTikTokAudio",
    "muteAudio",
    "reduceTransparency",
    "disableAnimations",
    "disableMetadata",
    "twitterGif",
+    "plausible_ignore"
 ];
 const exceptions = { // used for mobile devices
    "vQuality": "720"
@ -369,13 +369,11 @@ async function download(url) {
    if (sGet("vimeoDash") === "true") req.vimeoDash = true;
    if (sGet("audioMode") === "true") {
        req.isAudioOnly = true;
-        req.isNoTTWatermark = true; // video tiktok no watermark
        if (sGet("fullTikTokAudio") === "true") req.isTTFullAudio = true; // audio tiktok full
    } else {
        req.vQuality = sGet("vQuality").slice(0, 4);
        if (sGet("muteAudio") === "true") req.isAudioMuted = true;
        if (url.includes("youtube.com/") || url.includes("/youtu.be/")) req.vCodec = sGet("vCodec").slice(0, 4);
-        if ((url.includes("tiktok.com/") || url.includes("douyin.com/")) && sGet("disableTikTokWatermark") === "true") req.isNoTTWatermark = true;
    }

    if (sGet("disableMetadata") === "true") req.disableMetadata = true;
@ -566,7 +564,12 @@ function loadSettings() {
        eid("cobalt-body").classList.add('no-animation');
    }
    for (let i = 0; i < checkboxes.length; i++) {
-        if (sGet(checkboxes[i]) === "true") eid(checkboxes[i]).checked = true;
+        try {
+            if (sGet(checkboxes[i]) === "true") eid(checkboxes[i]).checked = true;
+        }
+        catch {
+            console.error(`checkbox ${checkboxes[i]} failed to initialize`)
+        }
    }
    for (let i in switchers) {
        changeSwitcher(i, sGet(i))
--- a/src/localization/languages/en.json
+++ b/src/localization/languages/en.json
@ -62,7 +62,6 @@
        "SettingsAudioFormatBest": "best",
        "SettingsAudioFormatDescription": "when \"best\" format is selected, you get audio the way it is on service's side. it's not re-encoded. everything else will be re-encoded.",
        "Keyphrase": "save what you love",
-        "SettingsRemoveWatermark": "disable watermark",
        "ErrorPopupCloseButton": "got it",
        "ErrorLengthAudioConvert": "i can't convert audio longer than {s} minutes. pick \"best\" format if you want to avoid limitations!",
        "SettingsAudioFullTikTok": "full audio",
@ -156,6 +155,10 @@
        "SettingsTwitterGifDescription": "converting looping videos to .gif reduces quality and majorly increases file size. if you want best efficiency, keep this setting off.",
        "ErrorTweetProtected": "this tweet is from a private account, so i can't see it. try another one!",
        "ErrorTweetNSFW": "this tweet contains sensitive content, so i can't see it. try another one!",
-        "UpdateEncryption": "encryption and new services"
+        "UpdateEncryption": "encryption and new services",
+        "PrivateAnalytics": "private analytics",
+        "SettingsDisableAnalytics": "opt out of private analytics",
+        "SettingsAnalyticsExplanation": "enable if you don't want to be included in anonymous traffic stats. read more about this in about > privacy policy (tl;dr: nothing about you is ever stored or tracked, no cookies are used).",
+        "AnalyticsDescription": "cobalt uses a self-hosted plausible instance to get an approximate number of how many people use it.\n\nplausible is fully compliant with GDPR, CCPA and PECR, doesn't use cookies, and never stores any identifiable info, not even your ip address.\n\nall data is aggregated and never personalized. nothing about what you download is ever saved anywhere. it's used just for anonymous traffic stats, nothing more.\n\nplausible is fully open source, just like cobalt, and if you want to learn more about it, you can do so <a class=\"text-backdrop link\" href=\"https://plausible.io\" target=\"_blank\">here</a>. if you wish to opt out of traffic stats, you can do it in settings > other."
    }
 }
--- a/src/localization/languages/ru.json
+++ b/src/localization/languages/ru.json
@ -62,7 +62,6 @@
        "SettingsAudioFormatBest": "лучший",
        "SettingsAudioFormatDescription": "когда выбран \"лучший\", ты получишь аудио без каких-либо изменений. такое, какое оно есть на стороне сервиса. если же выбрано что-то другое, то аудио будет немного сжато.",
        "Keyphrase": "сохраняй то, что любишь",
-        "SettingsRemoveWatermark": "убрать ватермарку",
        "ErrorPopupCloseButton": "ясно",
        "ErrorLengthAudioConvert": "я не могу конвертировать аудио дольше чем {s} минут(ы). выбери \"лучший\" формат, чтобы обойти ограничения.",
        "SettingsAudioFullTikTok": "полное аудио",
@ -158,6 +157,10 @@
        "SettingsTwitterGifDescription": "конвертирование зацикленного видео в .gif снижает качество и значительно увеличивает размер файла. если важна максимальная эффективность, то не используй эту функцию.",
        "ErrorTweetProtected": "этот твит из закрытого аккаунта, поэтому я не могу его увидеть. попробуй другой!",
        "ErrorTweetNSFW": "этот твит содержит деликатный контент, поэтому я не могу его увидеть. попробуй другой!",
-        "UpdateEncryption": "шифрование и новые сервисы"
+        "UpdateEncryption": "шифрование и новые сервисы",
+        "PrivateAnalytics": "приватная аналитика",
+        "SettingsDisableAnalytics": "отключить приватную аналитику",
+        "SettingsAnalyticsExplanation": "включи, если не хочешь быть частью анонимной статистики трафика. подробнее об этом можно прочитать в политике конфиденциальности (tl;dr: ничего о тебе или твоих действиях не хранится и не отслеживается, даже куки нет).",
+        "AnalyticsDescription": "кобальт использует собственный инстанс plausible чтобы иметь приблизительное представление о том, сколько людей им пользуются.\n\nplausible полностью соответствует GDPR, CCPA и PECR, не использует куки и никогда не хранит никакой идентифицируемой информации, даже ip-адрес.\n\nвсе данные агрегируются и никогда не персонализируются. ничего о том, что ты скачиваешь, никогда не сохраняется. это просто анонимная статистика трафика, ничего больше.\n\nplausible также как и кобальт имеет открытый исходный код, и, если ты хочешь узнать о нём больше, то это можно сделать <a class=\"text-backdrop link\" href=\"https://plausible.io\" target=\"_blank\">здесь</a>. а если же ты хочешь исключить себя из статистики, то это можно сделать в настройках > другое."
    }
 }
--- a/src/modules/pageRender/page.js
+++ b/src/modules/pageRender/page.js
@ -74,6 +74,14 @@ export default function(obj) {
        <link rel="preload" href="fonts/notosansmono.css" as="style">
        <link rel="preload" href="assets/meowbalt/error.png" as="image">
        <link rel="preload" href="assets/meowbalt/question.png" as="image">
+
+        ${process.env.PLAUSIBLE_HOSTNAME ?
+            `<script 
+                defer 
+                data-domain="${new URL(process.env.WEB_URL).hostname}" 
+                src="https://${process.env.PLAUSIBLE_HOSTNAME}/js/script.js"
+            ></script>`
+        : ''}
    </head>
    <body id="cobalt-body" ${platform === "d" ? 'class="desktop"' : ''}>
        <noscript>
@ -160,7 +168,9 @@ export default function(obj) {
                        }, {
                            name: "privacy",
                            title: `${emoji("🔒")} ${t("CollapsePrivacy")}`,
-                            body: t("PrivacyPolicy")
+                            body: t("PrivacyPolicy") + `${
+                                process.env.PLAUSIBLE_HOSTNAME ? `<br><br>${t("AnalyticsDescription")}` : ''
+                            }`
                        }, {
                            name: "legal",
                            title: `${emoji("📑")} ${t("CollapseLegal")}`,
@ -328,15 +338,6 @@ export default function(obj) {
                        }]
                    })
                })
-                + settingsCategory({
-                    name: "tiktok-watermark",
-                    title: "tiktok",
-                    body: checkbox([{
-                        action: "disableTikTokWatermark",
-                        name: t("SettingsRemoveWatermark"),
-                        padding: "no-margin"
-                    }])
-                })
                + settingsCategory({
                    name: "twitter",
                    title: "twitter",
@ -497,6 +498,21 @@ export default function(obj) {
                        padding: "no-margin"
                    }])
                })
+                + (() => {
+                    if (process.env.PLAUSIBLE_HOSTNAME) {
+                        return settingsCategory({
+                            name: "privacy",
+                            title: t('PrivateAnalytics'),
+                            body: checkbox([{
+                                action: "plausible_ignore",
+                                name: t("SettingsDisableAnalytics"),
+                                padding: "no-margin"
+                            }])
+                            + explanation(t('SettingsAnalyticsExplanation'))
+                        })
+                    }
+                    return ''
+                })()
                + settingsCategory({
                    name: "miscellaneous",
                    title: t('Miscellaneous'),
--- a/src/modules/processing/match.js
+++ b/src/modules/processing/match.js
@ -89,7 +89,6 @@ export default async function(host, patternMatch, url, lang, obj) {
                    host: host,
                    postId: patternMatch.postId,
                    id: patternMatch.id,
-                    noWatermark: obj.isNoTTWatermark,
                    fullAudio: obj.isTTFullAudio,
                    isAudioOnly: isAudioOnly
                });
--- a/src/modules/processing/services/tiktok.js
+++ b/src/modules/processing/services/tiktok.js
@ -58,7 +58,9 @@ export default async function(obj) {
    detail = selector(detail, obj.host, postId);
    if (!detail) return { error: 'ErrorCouldntFetch' };

-    let video, videoFilename, audioFilename, isMp3, audio, images, filenameBase = `${obj.host}_${postId}`;
+    let video, videoFilename, audioFilename, isMp3, audio, images,
+        filenameBase = `${obj.host}_${detail.author.unique_id}_${postId}`;
+
    if (obj.host === "tiktok") {
        images = detail.image_post_info ? detail.image_post_info.images : false
    } else {
@ -66,14 +68,10 @@ export default async function(obj) {
    }

    if (!obj.isAudioOnly && !images) {
-        video = obj.host === "tiktok" ? detail.video.download_addr.url_list[0] : detail.video.play_addr.url_list[2].replace("/play/", "/playwm/");
-        videoFilename = `${filenameBase}_video.mp4`;
-        if (obj.noWatermark) {
-            video = obj.host === "tiktok" ? detail.video.play_addr.url_list[0] : detail.video.play_addr.url_list[0];
-            videoFilename = `${filenameBase}_video_nw.mp4` // nw - no watermark
-        }
+        video = detail.video.play_addr.url_list[0];
+        videoFilename = `${filenameBase}.mp4`;
    } else {
-        let fallback = obj.host === "douyin" ? detail.video.play_addr.url_list[0].replace("playwm", "play") : detail.video.play_addr.url_list[0];
+        let fallback = detail.video.play_addr.url_list[0];
        audio = fallback;
        audioFilename = `${filenameBase}_audio_fv`;  // fv - from video
        if (obj.fullAudio || fallback.includes("music")) {
--- a/src/modules/stream/types.js
+++ b/src/modules/stream/types.js
@ -4,7 +4,6 @@ import { ffmpegArgs, genericUserAgent } from "../config.js";
 import { metadataManager } from "../sub/utils.js";
 import { request } from "undici";
 import { create as contentDisposition } from "content-disposition-header";
-import { AbortController } from "abort-controller"

 function closeRequest(controller) {
    try { controller.abort() } catch {}
--- a/src/modules/sub/utils.js
+++ b/src/modules/sub/utils.js
@ -9,7 +9,7 @@ const apiVar = {
        aFormat: ["best", "mp3", "ogg", "wav", "opus"],
        filenamePattern: ["classic", "pretty", "basic", "nerdy"]
    },
-    booleanOnly: ["isAudioOnly", "isNoTTWatermark", "isTTFullAudio", "isAudioMuted", "dubLang", "vimeoDash", "disableMetadata", "twitterGif"]
+    booleanOnly: ["isAudioOnly", "isTTFullAudio", "isAudioMuted", "dubLang", "vimeoDash", "disableMetadata", "twitterGif"]
 }
 const forbiddenCharsString = ['}', '{', '%', '>', '<', '^', ';', '`', '$', '"', "@", '='];

@ -79,7 +79,6 @@ export function checkJSONPost(obj) {
        aFormat: "mp3",
        filenamePattern: "classic",
        isAudioOnly: false,
-        isNoTTWatermark: false,
        isTTFullAudio: false,
        isAudioMuted: false,
        disableMetadata: false,
--- a/src/test/tests.json
+++ b/src/test/tests.json
@ -562,125 +562,10 @@
            "status": "error"
        }
    }],
-    "douyin": [{
-        "name": "short link video, with watermark",
-        "url": "https://v.douyin.com/2p4Aya7/",
-        "params": {},
-        "expected": {
-            "code": 200,
-            "status": "stream"
-        }
-    }, {
-        "name": "short link video (isNoTTWatermark)",
-        "url": "https://v.douyin.com/2p4Aya7/",
-        "params": {
-            "isNoTTWatermark": true
-        },
-        "expected": {
-            "code": 200,
-            "status": "stream"
-        }
-    }, {
-        "name": "short link video (isAudioOnly)",
-        "url": "https://v.douyin.com/2p4Aya7/",
-        "params": {
-            "isAudioOnly": true
-        },
-        "expected": {
-            "code": 200,
-            "status": "stream"
-        }
-    }, {
-        "name": "short link video (isAudioOnly, isTTFullAudio)",
-        "url": "https://v.douyin.com/2p4Aya7/",
-        "params": {
-            "isAudioOnly": true,
-            "isTTFullAudio": true
-        },
-        "expected": {
-            "code": 200,
-            "status": "stream"
-        }
-    }, {
-        "name": "long link video (isNoTTWatermark)",
-        "url": "https://www.douyin.com/video/7120601033314716968",
-        "params": {
-            "isNoTTWatermark": true
-        },
-        "expected": {
-            "code": 200,
-            "status": "stream"
-        }
-    }, {
-        "name": "images",
-        "url": "https://v.douyin.com/MdVwo31/",
-        "params": {},
-        "expected": {
-            "code": 200,
-            "status": "picker"
-        }
-    }, {
-        "name": "long link inexistent",
-        "url": "https://www.douyin.com/video/7120851458451417478",
-        "params": {},
-        "expected": {
-            "code": 400,
-            "status": "error"
-        }
-    }, {
-        "name": "short link inexistent",
-        "url": "https://v.douyin.com/2p4ewa7/",
-        "params": {},
-        "expected": {
-            "code": 400,
-            "status": "error"
-        }
-    }],
    "tiktok": [{
-        "name": "short link (vt) video, with watermark",
-        "url": "https://vt.tiktok.com/ZS85U86aa/",
-        "params": {},
-        "expected": {
-            "code": 200,
-            "status": "stream"
-        }
-    }, {
-        "name": "short link (vt) video (isNoTTWatermark)",
-        "url": "https://vt.tiktok.com/ZS85U86aa/",
-        "params": {
-            "isNoTTWatermark": true
-        },
-        "expected": {
-            "code": 200,
-            "status": "stream"
-        }
-    }, {
-        "name": "short link (vm) video (isAudioOnly)",
-        "url": "https://vm.tiktok.com/ZMYrYAf34/",
-        "params": {
-            "isAudioOnly": true
-        },
-        "expected": {
-            "code": 200,
-            "status": "stream"
-        }
-    }, {
-        "name": "short link (vm) video (isAudioOnly, isTTFullAudio)",
-        "url": "https://vm.tiktok.com/ZMYrYAf34/",
-        "params": {
-            "isAudioOnly": true,
-            "isTTFullAudio": true
-        },
-        "expected": {
-            "code": 200,
-            "status": "stream"
-        }
-    }, {
-        "name": "long link video (isNoTTWatermark)",
+        "name": "long link video",
        "url": "https://www.tiktok.com/@fatfatmillycat/video/7195741644585454894",
-        "params": {
-            "isNoTTWatermark": true
-        },
+        "params": {},
        "expected": {
            "code": 200,
            "status": "stream"