Support compression for Actions logs to save storage space and
bandwidth. Inspired by
https://github.com/go-gitea/gitea/issues/24256#issuecomment-1521153015
The biggest challenge is that the compression format should support
[seekable](https://github.com/facebook/zstd/blob/dev/contrib/seekable_format/zstd_seekable_compression_format.md).
So when users are viewing a part of the log lines, Gitea doesn't need to
download the whole compressed file and decompress it.
That means gzip cannot help here. And I did research, there aren't too
many choices, like bgzip and xz, but I think zstd is the most popular
one. It has an implementation in Golang with
[zstd](https://github.com/klauspost/compress/tree/master/zstd) and
[zstd-seekable-format-go](https://github.com/SaveTheRbtz/zstd-seekable-format-go),
and what is better is that it has good compatibility: a seekable format
zstd file can be read by a regular zstd reader.
This PR introduces a new package `zstd` to combine and wrap the two
packages, to provide a unified and easy-to-use API.
And a new setting `LOG_COMPRESSION` is added to the config, although I
don't see any reason why not to use compression, I think's it's a good
idea to keep the default with `none` to be consistent with old versions.
`LOG_COMPRESSION` takes effect for only new log files, it adds `.zst` as
an extension to the file name, so Gitea can determine if it needs
decompression according to the file name when reading. Old files will
keep the format since it's not worth converting them, as they will be
cleared after #31735.
<img width="541" alt="image"
src="https://github.com/user-attachments/assets/e9598764-a4e0-4b68-8c2b-f769265183c9">
(cherry picked from commit 33cc5837a655ad544b936d4d040ca36d74092588)
Conflicts:
assets/go-licenses.json
go.mod
go.sum
resolved with make tidy
- We were previously using `github.com/keybase/go-crypto`, because the
package for openpgp by Go itself is deprecated and no longer
maintained. This library provided a maintained version of the openpgp
package. However, it hasn't seen any activity for the last five years,
and I would therefore consider this also unmaintained.
- This patch switches the package to `github.com/ProtonMail/go-crypto`
which provides a maintained version of the openpgp package and was
already being used in the tests.
- Adds unit tests, I've carefully checked the callstacks to ensure the
OpenPGP-related code was covered under either a unit test or integration
tests to avoid regression, as this can easily turn into security
vulnerabilities if a regression happens here.
- Small behavior update, revocations are now checked correctly instead
of checking if they merely exist and the expiry time of a subkey is used
if one is provided (this is just cosmetic and doesn't impact security).
- One more dependency eliminated :D
- Update the `github.com/santhosh-tekuri/jsonschema` library from v5 to
v6.
- Update the migration loader function to a type, which is now required
in V6.
- `github.com/santhosh-tekuri/jsonschema/v6` was already used by gof3,
so removing the v5 library and using the v6 library reduces the binary
size of Forgejo.
- Before: 95912040 bytes
- After: 95706152 bytes
- Go's deadcode eliminator is quite simple, if you put a public function
in a package `aa/bb` that is used only by tests, it would still be built
if package `aa/bb` was imported. This means that if such functions use
libraries relevant only to tests that those libraries would still be
be built and increase the binary size of a Go binary.
- This is also the case with Forgejo, `models/migrations/base/tests.go`
contained functions exclusively used by tests which (skipping some steps
here) imports https://github.com/ClickHouse/clickhouse-go, which is
2MiB. The `code.gitea.io/gitea/models/migrations/base` package is
imported by `cmd/doctor` and thus the code of the clickhouse library is
also built and included in the Forgejo binary, although entirely unused
and not reachable.
- This patch moves the test-related functions to their own package, so
Go's deadcode eliminator knows not to build the test-related functions
and thus reduces the size of the Forgejo binary.
- It is not possible to move this to a `_test.go` file because Go does
not allow importing functions from such files, so any test helper
function must be in a non-test package and file.
- Reduction of size (built with `TAGS="sqlite sqlite_unlock_notify" make
build`):
- Before: 95912040 bytes (92M)
- After: 92306888 bytes (89M)
- Remove a unused dependency. This dependency was added to handle YAML
'frontmatter' meta, parsing them and converting them to a table or
details in the resulting HTML. As can be read in the issue that reported
the behavior of YAML frontmatter being rendered literally,
https://github.com/go-gitea/gitea/issues/5377.
- It's an unused dependency as the codebase since then moved on to do this YAML
parsing and rendering on their own, this was implemented in
812cfd0ad9.
- Adds unit tests that was related to this functionality, to proof the
codebase already handles this and to prevent regressions.
This PR contains the following updates:
| Package | Type | Update | Change |
|---|---|---|---|
| code.forgejo.org/f3/gof3/v3 | require | minor | `v3.3.1` -> `v3.4.0` |
---
### Configuration
📅 **Schedule**: Branch creation - "before 4am" (UTC), Automerge - "before 4am" (UTC).
🚦 **Automerge**: Disabled by config. Please merge this manually once you are satisfied.
♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox.
🔕 **Ignore**: Close this PR and you won't be reminded about this update again.
---
- [ ] <!-- rebase-check -->If you want to rebase/retry this PR, check this box
---
This PR has been generated by [Renovate Bot](https://github.com/renovatebot/renovate).
<!--renovate-debug:eyJjcmVhdGVkSW5WZXIiOiIzNy40MDkuMSIsInVwZGF0ZWRJblZlciI6IjM3LjQwOS4xIiwidGFyZ2V0QnJhbmNoIjoiZm9yZ2VqbyIsImxhYmVscyI6WyJkZXBlbmRlbmN5LXVwZ3JhZGUiXX0=-->
Co-authored-by: Twenty Panda <twenty-panda@posteo.com>
Reviewed-on: https://codeberg.org/forgejo/forgejo/pulls/4196
Reviewed-by: Earl Warren <earl-warren@noreply.codeberg.org>
Co-authored-by: Renovate Bot <forgejo-renovate-action@forgejo.org>
Co-committed-by: Renovate Bot <forgejo-renovate-action@forgejo.org>
replace reply with forgejos forked version
If plain text is selected as the message format in e.g. Apple Mail, the inline attachments are no longer at the end of the mail, but instead directly where they are in the mail. When parsing the mail, these inline attachments are replaced by "--". The new reply version no longer cuts the text at the first "--".
Tests for this are present in reply (7dc5750c6d).
Fixes https://codeberg.org/forgejo/forgejo/issues/3496#issuecomment-1798416
---
Additionally, I reduced the allocations for the inline attachments.
Reviewed-on: https://codeberg.org/forgejo/forgejo/pulls/3747
Reviewed-by: Earl Warren <earl-warren@noreply.codeberg.org>
Co-authored-by: Beowulf <beowulf@beocode.eu>
Co-committed-by: Beowulf <beowulf@beocode.eu>
- It was only used to parse old U2F data to webauthn credentials. We
only used the public key and keyhandle. This functiontionality was
reworked to `parseU2FRegistration`.
- Tests are already present, `Test_RemigrateU2FCredentials`.
- Remove `gitea.com/lunny/dingtalk_webhook` as dependency, we only use
two structs which are small enough to be recreated in Forgejo and don't
need to rely on the dependency.
- Existing tests (thanks @oliverpool) prove that this has no effect.
- Update github.com/bufbuild/connect-go to
https://github.com/connectrpc/connect-go.
- This is a fork that's actively maintained and is recommend by the
original library. Looking at the recent release notes, it looks like
going in the right direction what one would expect of a library, no
strange features being added, lots of improvements.
- There's still an indirect dependency by
`code.gitea.io/actions-proto-go` on a old version of `connect-go`.
- https://github.com/NYTimes/gziphandler doesn't seems to be maintained
anymore and Forgejo already includes
https://github.com/klauspost/compress which provides a maintained and
faster gzip handler fork.
- Enables Jitter to prevent BREACH attacks, as this *seems* to be
possible in the context of Forgejo.
(cherry picked from commit cc2847241d82001babd8d40c87d03169f21c14cd)
(cherry picked from commit 99ba56a8761dd08e08d9499cab2ded1a6b7b970f)
Conflicts:
go.sum
https://codeberg.org/forgejo/forgejo/pulls/1581
(cherry picked from commit 711638193daa2311e2ead6249a47dcec47b4e335)
(cherry picked from commit 9c12a37fde6fa84414bf332ff4a066facdb92d38)
(cherry picked from commit d13065345431a499f9e0b7a3c2043d7487b8aa5b)
(cherry picked from commit 45a16f8c3c6f7d5e4aab8fdde6a621cf36e4801c)
(cherry picked from commit a497acb31f76d580c8b0567f9461274bd78080f4)
(cherry picked from commit fe87fd828973945192b98310c5c3b2001c6e0f86)
(cherry picked from commit 6ac12e6693cf45cb12109028dabd868957c4b74c)
(cherry picked from commit 981ec37e1e72ab19c20067ff4d2a7e20a60d3305)
(cherry picked from commit 5d6892ec10086f0ba63f26693faa82d0fd4e3f4a)
(cherry picked from commit 9df7968f4fc72de9788d84ca3f349e4c98ee630e)
(cherry picked from commit 7d588d183329cd760053663ea2e1e82e62958409)
Conflicts:
routers/web/web.go
https://codeberg.org/forgejo/forgejo/pulls/2075
(cherry picked from commit defb101281f5a6ba410abc763674bafa7b63dffd)
(cherry picked from commit 5830f204a17767fda3e45d16dbf3af8c32e7f387)
(cherry picked from commit 029f4e98636a7776f430684e9d7142d69a444f96)
(cherry picked from commit 816fe558126d0ecce969fdf2a196fa6afdcca792)
Conflicts:
go.sum
https://codeberg.org/forgejo/forgejo/pulls/2249
(cherry picked from commit 99866d804560b415b6158371eb0efd17d097cfe0)
(cherry picked from commit f42622c7d5a28859f535e0d86ece06101baad1ef)
(cherry picked from commit a39e7f2a79f6527d45439f21bd88378264160c3a)
(cherry picked from commit afa2a31bb99c0fd9cd25c3c0279e6c49695b1900)
(cherry picked from commit 276e8856e594ad7e73414382d34fdf0278cbca6a)
(cherry picked from commit 68e3bd469f2e8190db70d4e1564fb46d01feb5f4)
(cherry picked from commit af124b9ccbb0b699ea5d1bf1530613cb9a96f205)
(cherry picked from commit b89ab4874d403c784c92e579f4f6a854621c0078)
(cherry picked from commit 0f2a2f0d0ff9851428d6899e307f3547d7651f87)
(cherry picked from commit 80999363c73e4e01cbf9116491743c87baa952af)
(cherry picked from commit f8880b5463aa1db047d89080d21e7a69979eb4a6)
(cherry picked from commit 5f4cf4f6e143237c81da3e80875727cef4b76343)
(cherry picked from commit b38e26bc1a8bb117f571672961a8445cfc02c953)
(cherry picked from commit d839e0033244b63df16c8b548cc52106fc693629)
(cherry picked from commit 32ffe2e4f12c16a44a759c906078d40731c1a0c7)
(cherry picked from commit f1fd0504add78ae3fb5c710ab5a9ada20321afac)
(cherry picked from commit 6d77ea4d60a193d9d4175c4943b34077228e5964)
(cherry picked from commit 61a0a4a276303c3fd56f93fd109e3416cfdf7c60)
(cherry picked from commit a90b4126fdd55feef43f45d1dc3e3400806a476b)
(cherry picked from commit 9a20538fb4e88682e213a20813b77cc0f602fbfd)
(cherry picked from commit ce0fc02f0fbaa45b146fed175ce68bd02c507f3e)
(cherry picked from commit 541f7cb026f976d078ecb6da3a6c9e13cc4336f2)
(cherry picked from commit d6d0c2ab78a14d7aac8d7b6b0d007149de2f7295)
(cherry picked from commit 2c28f5ad2496cf30eb15d6caf9171b79e5017141)
(cherry picked from commit 9571bddb3308e3c1f0383e60f972ca61a0a467b7)
(cherry picked from commit c83ba08d01f149ecd52d983eec76bd60822c1ddd)
(cherry picked from commit 30e7d567ede79c015d0d115d9a2d535e6c681cb9)
(cherry picked from commit a8b8c3eba75511449dc97fa27b37db1076ce95f0)
(cherry picked from commit 8e053e1ade4710ade3b6e4bc6aa04fe9281243d7)
(cherry picked from commit 9e3b0f7520a56e5eb22cd0e33e231ea9063f0e1f)
(cherry picked from commit 2343b9bd09cda3e474b36842fae419f9fe32b134)
(cherry picked from commit 56572d4156050c2beb62f63a871375fdc2424271)
(cherry picked from commit 9b09eda1680282f8114f752e52afd544ef30350c)
(cherry picked from commit 86a8b7b4904158ea80d259d7a1846528a9b3c403)
(cherry picked from commit 99a550c0e3de3bd8d17a610b849da0c04f776dbd)
(cherry picked from commit bdcf3f51e07c9f5f067fca7501ad69c42f925197)
(cherry picked from commit cec8f2d31e1cd9237120ee57af80b17e4abf026e)
(cherry picked from commit 25aa22ba2b8b8bdb215c14002ebc137e2e70cad4)
(cherry picked from commit 31510249a01fad40cc001277f5bcbe57248b0330)
(cherry picked from commit 95dc569227ac57c2efe7817edb6749fcead0ec24)
(cherry picked from commit f6caf5f1b9f8f560fcfef29247525c018976dcde)
(cherry picked from commit aba34fbf70dce183907083593cab7716597575a3)
(cherry picked from commit 41b816fdac30cfca311d5c24c155301de5d08a40)
(cherry picked from commit c98e79b89f5a12b50b6eb1ff1d0afc2cbd756ca3)
(cherry picked from commit d33d4f193cd79493e3d206e0c73395aea5bd305c)
(cherry picked from commit 4e5bb41cbe9823f8c198a54b56f0ee66b9cf8bf8)
(cherry picked from commit 3aa8ddb8cb4cb836b98c465342b106d027868606)
(cherry picked from commit e8057040bb751d1bdcb0e7412495da353189a02f)
(cherry picked from commit f3bf61e51e8c44ac098dca03532507bae5b65fc1)
(cherry picked from commit e9d08aad76cc38a81e9489871e1116819f84052e)
- The [rupture](https://github.com/ethantkoenig/rupture) dependency was
essentially outdated in the sense it was using old version of
dependencies.
- The usage by Forgejo was rather a small portion, so that portion is
now vendored (with its tests).
- Removes old dependencies from go.sum (less dependencies is better for
reviewing what the heck we're importing). Just to note that they were
likely not being used by Go's build process (according to
https://go.dev/ref/mod#minimal-version-selection), so it's really a
matter of formal cleaning up dependencies we don't use and therefor
don't want to download and be in our go.sum.
(cherry picked from commit aa72a5f009b5027b2324106343f91b466ba46293)
Conflicts:
go.sum
https://codeberg.org/forgejo/forgejo/pulls/2148
(cherry picked from commit fbe8d65f0b1836b2e771991b4d5d12f1bfa938ed)
(cherry picked from commit e18debcb6a9476f60d364e847265b4ac7fb76c8e)
Conflicts:
go.sum
https://codeberg.org/forgejo/forgejo/pulls/2245
(cherry picked from commit 8c43c2ada82102a0df44fd874c4f5fe3a36ef758)
Update golang.org/x/crypto for CVE-2023-48795 and update other packages.
`go-git` is not updated because it needs time to figure out why some
tests fail.
Replace `github.com/gogs/cron` with `github.com/go-co-op/gocron` as the
former package is not maintained for many years.
---------
Co-authored-by: delvh <dev.lh@web.de>
Replace #10912
And there are many new tests to cover the CLI behavior
There were some concerns about the "option order in hook scripts"
(https://github.com/go-gitea/gitea/pull/10912#issuecomment-1137543314),
it's not a problem now. Because the hook script uses `/gitea hook
--config=/app.ini pre-receive` format. The "config" is a global option,
it can appear anywhere.
----
## ⚠️ BREAKING ⚠️
This PR does it best to avoid breaking anything. The major changes are:
* `gitea` itself won't accept web's options: `--install-port` / `--pid`
/ `--port` / `--quiet` / `--verbose` .... They are `web` sub-command's
options.
* Use `./gitea web --pid ....` instead
* `./gitea` can still run the `web` sub-command as shorthand, with
default options
* The sub-command's options must follow the sub-command
* Before: `./gitea --sub-opt subcmd` might equal to `./gitea subcmd
--sub-opt` (well, might not ...)
* After: only `./gitea subcmd --sub-opt` could be used
* The global options like `--config` are not affected
The package `github.com/nfnt/resize` is deprecated and archived by the
author. `github.com/oliamb/cutter` is not maintained since 2018. We
could use `golang.org/x/image/draw` instead.
Bumping `github.com/golang-jwt/jwt` from v4 to v5.
`github.com/golang-jwt/jwt` v5 is bringing some breaking changes:
- standard `Valid()` method on claims is removed. It's replaced by
`ClaimsValidator` interface implementing `Validator()` method instead,
which is called after standard validation. Gitea doesn't seem to be
using this logic.
- `jwt.Token` has a field `Valid`, so it's checked in `ParseToken`
function in `services/auth/source/oauth2/token.go`
---------
Co-authored-by: Giteabot <teabot@gitea.io>
