* Tidy up emoji parsing on profile submission
Don't bother reparsing for emoji unless one of the fields that can have
emoji in it has changed.
Deduplicate emoji between the display name and profile note - I'm not
sure whether this was hurting anything, but better safe.
* Deduplicate emoji when parsing remote accounts
Some servers - Misskey at least - don't deduplicate emoji, so it's
possible to get an account which has the same emoji used in both the
display name and note and therefore includes that emoji twice in its
metadata. When we start trying to put those into our database, we run
into a uniqueness constraint and fall over.
This change just deduplicates at the point of construction of an
account.
This moves checking if the request is valid as early as possible in the
chain. This should ensure that for an invalid request we never bother
acquiring the wait queue and taking up a spot in it.
* improve error readability, mark "bad hosts" as fastFail
Signed-off-by: kim <grufwub@gmail.com>
* pull in latest go-byteutil version with byteutil.Reader{}
Signed-off-by: kim <grufwub@gmail.com>
* use rewindable body reader for post requests
Signed-off-by: kim <grufwub@gmail.com>
---------
Signed-off-by: kim <grufwub@gmail.com>
This updates the middleware log.WithField calls that create new loggers
to include the context the first time around. Without it the requestID
does not get logged.
Fixup from #1476
This adds a lightweight form of tracing to GTS. Each incoming request is
assigned a Request ID which we then pass on and log in all our log
lines. Any function that gets called downstream from an HTTP handler
should now emit a requestID=value pair whenever it logs something.
Co-authored-by: kim <grufwub@gmail.com>
* [bugfix] set cache-control max-age dynamically for s3
* woops
* double whoops
* time until, thank you linter, bless you, you're the best, no matter what kim says
* aa
Have a separate template for:
* Frontend, with labels bug and frontend and requiring the browser field
* Regular/backend, for other bugs with only the bug label and removal of
the browser field
* [bug] Fix nginx fileserver caching example
This updates the example to ensure the nginx proxies the request on to
GTS if the file is not found on disk. This can happen due to media
pruning.
* [chore] Set cache-control in nginx to private
This makes the header match with the backend. For things from the
fileserver it may not be appropriate for anything other than a private
cache (i.e the client) to cache things.
* start fiddling with media + account queries a little
* initialize state when pruning
* allow for unsetting remote media
make sure to wait til media loaded
fix silly tiny bug
* move comment a bit for readability
* slight reformat of fetchRemoteAccount{Avatar,Header}
* fix issue after rebase
* slightly neaten up logic of avatar/header media handling
* remove log prefix (callername log field handles this)
---------
Signed-off-by: kim <grufwub@gmail.com>
Co-authored-by: kim <grufwub@gmail.com>
* replace media workers with just runners.WorkerPool, move to state structure, use go-sched for global task scheduling
* improved code comment
* fix worker tryUntil function, update go-runners/go-sched
* make preprocess functions package public, use these where possible to stop doubled up processing
* remove separate emoji worker pool
* limit calls to time.Now() during media preprocessing
* use Processor{} to manage singular runtime of processing media
* ensure workers get started when media manager is used
* improved error setting in processing media, fix media test
* port changes from processingmedia to processing emoji
* finish code commenting
* finish code commenting and comment-out client API + federator worker pools until concurrency worker pools replaced
* linterrrrrrrrrrrrrrrr
---------
Signed-off-by: kim <grufwub@gmail.com>
* remove throttling timers, support setting retry-after, use retry-after in transport
* remove unused variables
* add throttling-retry-after to cmd flags
* update envparsing to include new throttling-retry-after
* update example config to include retry-after documentation
* also support retry-after formatted as date-time, ensure max backoff time
---------
Signed-off-by: kim <grufwub@gmail.com>
* small formatting changes, rewrite fetchRemoteMedia to use separate funcs + use mutex lock correctly
* move url parsing before acquiring mutex locks
* use wrapped mutexes to allow safe unlocking. (previously i did a fucky and passed mutex by value...)
* remove unused code
* use consistent map keying for dereferencing headers/avatars
---------
Signed-off-by: kim <grufwub@gmail.com>
* serve publickey separately from AP, don't throttle it
* update nginx cache documentation, cache main-key too
* throttle public key, but separately from other endpoints
This adds a line to the bug report template to indicate that
compatibility issues with closed source clients should be reported to
developers of that client instead.
