NZB Refresh X (NZBreX)

NZB Refresh X (NZBreX) is a command-line tool to restore missing Usenet articles by downloading them from providers where they are still available and re-uploading them to others.

This tool is a modernized fork of Tensai75/nzbrefresh.

---

What It Does

NZBreX loads an NZB file (given as a command-line argument) and checks the availability of each article/segment across all configured Usenet providers (from provider.json).

If it finds articles missing from one or more providers but still available on at least one other, it will:

1. Download the missing article from a provider where it is present.
2. Cache the article locally (optional, depending on your config).
3. Re-upload the article to one or more providers where it was missing.

The tool uses the NNTP commands:

• STAT for checking
• ARTICLE for downloading
• IHAVE or POST for uploading

NZBreX runs in one of these modes:

Mode	Start with
One NZB	-nzb file.nzb	check, download, re-upload, then exit
Directory	-nzbdir /path	process every NZB that appears in a directory
Web interface	-web	upload NZBs in the browser and run them as a queue, see Web interface
NNTP proxy	-proxytcp / -proxytls	serve articles from your providers to a newsreader, see NNTP proxy

Only one mode per process.

---

Multiplexing / Grouping Provider Accounts

• You can group multiple accounts; articles will only be uploaded once per group.
• If provider.Group is not set, each provider with an empty group creates a group by provider.Name.
• Check, Download, and Upload can run concurrently or sequentially:
  • Check first (-checkfirst), then Download and Upload concurrently
  • Or with -uploadlater (TODO): Download only, then Upload later
  • Set NoUpload to skip uploading but allow download into cache.
• Upload priorities follow the order in provider.json.

---

Behavior

• For every connection, 1 go GoWorker(...) is spawned with 3 private goroutines (Check, Down, Reup).

• The article body is uploaded exactly as originally received.

• Some non-essential headers may be removed during re-upload:

  cleanHeader = []string{
    "X-",
    "Date:",
    "Nntp-",
    "Path:",
    "Xref:",
    "Cancel-",
    "Injection-",
    "User-Agent:",
    "Organization:",
  }

• The Date header is updated to the current date.

• Once successfully uploaded to one provider, the article should propagate to others.

---

Cache

• The cache is a folder with subfolders for each NZB, holding downloaded segment.art files.
• The subfolder is the sha256 hash of the nzb file; .art files are hashed by <messageID@xyz> (including < and >).
• To fill your cache, set all providers to 'NoUpload': true.
• The cache is checked before download requests; uploads are always fed from cache if available.
• DL/UL queues can rise and fall as items are checked and processed.
• Warning: the -cc (Check Cache on Boot) flag can be hard on traditional hard drives if -crw is set high (default 100).

---

Installation

1. Download pre-built binaries from the Releases page (the release notes say which file to take), or

2. Compile from source (see Build Instructions below)

3. Configure provider.json with your Usenet provider details.

4. Run the program from the command line:

   ./nzbrex -checkonly -nzb nzbs/ubuntu-24.04-live-server-amd64.iso.nzb.gz
   ./nzbrex -cd cache -checkfirst -nzb nzbs/ubuntu-24.04-live-server-amd64.iso.nzb.gz
   ./nzbrex --cd=cache --checkfirst --nzb=nzbs/ubuntu-24.04-live-server-amd64.iso.nzb.gz

Build Instructions

NZBreX builds in two variants. They differ only in the yEnc decoder used by -yenctest=4 (the default -crc32 mode):

Variant	Build	yEnc decoder (-yenctest=4)
purego (default)	go build .	go-yenc-simd, pure Go
rapidyenc	go build -tags rapidyenc .	rapidyenc's C code via cgo; -yencgo switches to go-yenc-simd

Pure Go (default)

go build .

No C compiler, no cgo. Cross-compiles to any target, e.g. CGO_ENABLED=0 GOOS=windows GOARCH=amd64 go build . or GOOS=darwin GOARCH=arm64. Built on Linux with CGO_ENABLED=0, the binary is static (runs on glibc and musl/Alpine). The purego release files are built this way: on Alpine take those, the rapidyenc Linux files need glibc 2.34 or newer and libstdc++.

For go-yenc-simd's AVX2/AVX-512 code on amd64, build with GOEXPERIMENT=simd go build . (Go 1.27; an experiment, outside the Go 1 compatibility promise). Without it the decoder still works, in plain Go.

With rapidyenc's C decoder

Needs a C/C++ compiler and cmake; the rapidyenc library is built first. This also works on musl (Alpine: apk add cmake g++ make linux-headers). Tested from source with both variants on AlmaLinux 9/10, Alpine 3.22–3.24, Arch, CentOS Stream 9/10, Debian 13, Gentoo, openEuler 25.03, openSUSE Leap 15.6/16.0, Rocky 9/10 and Ubuntu 24.04–26.04 (amd64). With gcc older than 8 (Leap 15.6 ships gcc 7) rapidyenc builds without its AVX-512 VBMI2 kernel.

# Linux
cd rapidyenc && ./build_rapidyenc_linux-amd64.sh && cd ..
go build -tags rapidyenc .

REM Windows, native: Go, MinGW-w64 and CMake installed, then
build_windows.bat

# Windows, cross-compiled from Linux
sudo apt install gcc-mingw-w64-x86-64 g++-mingw-w64-x86-64 cmake
./local_crossbuild_windows-amd64.sh

The Windows executable will be statically linked and only depend on core Windows system libraries.

---

Running NZBreX

Run with the following arguments:

• You can use single - or double -- for flags.
• -help prints the latest help with all arguments and info texts.

Main flags:

• -nzb string /path/file.nzb (default "test.nzb")
• -provider string /path/provider.json (default "provider.json")
• -cc [true|false] check NZB vs cache on boot (default: false)
• -cd string /path/to/cache/dir
• -crw int number of Cache Reader/Writer routines (default 100)
• -crc32 [true|false] check CRC32 of articles while downloading (default: false)
• -checkfirst [true|false] if false: start downs/reups as soon as segments are checked (default: false)
• -uploadlater (TODO) [true|false] needs cache! start uploads when everything is cached (default: false)
• -nzbdir /path watch this directory: process its .nzb/.nzb.gz files one at a time, oldest first. A file is taken once it is unchanged between two scans; afterwards it moves to done/ (or failed/ if the run couldn't start) in that directory. SIGINT/SIGTERM stops the running NZB, which stays in place for the next start. Not with -nzb, -web or the proxy.
• -dirrefresh seconds with -nzbdir: scan interval (default: 60, minimum 15)
• -checkonly [true|false] check online status only: no downs/reups (default: false)
• -verify [true|false] waits and tries to verify/recheck all reups (default: false)
• -verbose [true|false] more output (default: true)
• -discard [true|false] reduce console output to minimum (default: false)
• -log [true|false] logs to file (default: false)
• -bar [true|false] show progress bars (buggy) (default: false)
• -bug [true|false] full debug (default: false)
• -debug [true|false] part debug (default: false)
• -debugcache [true|false] (default: false)
• -printstats int print stats every N seconds. 0 is spammy, -1 disables output.
• -print430 [true|false] print notice about error code 430 (article not found)
• -slomoc int sleep N ms before checking
• -slomod int sleep N ms before downloading
• -slomou int sleep N ms before uploading
• -cleanhdr [true|false] remove unwanted headers (default: true)
• -cleanhdrfile load unwanted headers from file
• -maxartsize int limit article size (default 1048576)
• -mem int limit memory usage to N segments in RAM (0 = auto)
• -chansize int set internal channel size for check/down/reup queues
• -prof start cpu+mem profiler
• -profweb start profiling webserver at address
• -version print app version
• -benchmark measure how fast this machine decodes yEnc and exit: go-yenc-simd with every CPU kernel it can use, and in the rapidyenc build also the C decoder (-benchtime sets the time per value, default 1s)
• -yenccpu limit parallel decoding with -crc32=true (default: 8)
• -yencout [true|false] write yenc parts to cache (needs -cd)
• -yencmerge [true|false] merge yenc parts into target files
• -yencdelparts [true|false] delete .part.N.yenc files after merge (only with -yencmerge)
• -rapidyencbufsize "set only if you know what you do! (default: 4K) (experimental/testing)"
• -doublecheckrapidyenccrc "[true|false] (experimental/testing)"
• -yenctest select mode 1 (bytes), 2 (lines), or 3 (direct) or 4 (rapidyenc C or go-yenc-simd, see Build Instructions) for -crc32. Mode 2 uses less memory, 3 is experimental. (default: 4)
• -dlmissing [true|false] download the articles missing on any provider into the cache (-cd), also when no provider takes uploads, e.g. to re-upload them later. (default: false)
• -dlall [true|false] download every article not yet in the cache (-cd): fill the cache, or get the files with -crc32 -yencout -yencmerge. Without an upload provider this makes NZBreX a plain downloader. (default: false)
• -yencgo [true|false] with -yenctest=4 in a -tags rapidyenc build: decode with go-yenc-simd instead of the C code. Builds without the tag always use go-yenc-simd. (default: false)
• -yencasync limits async parallel decoding with -crc32=true and -yenctest=3 or 4. 0 defaults to runtime.NumCPU().
• -debugsharedcc debug sharedConn Chan (default: false)
• -debugflags debug item flags (default: false)
• -debugcr debug check routine (default: false)
• -debugdr debug downs routine (default: false)
• -debugur debug reups routine (default: false)
• -debugstat debug STAT (default: false)
• -debugarticle debug ARTICLE (default: false)
• -debugihave debug IHAVE (default: false)
• -debugpost debug POST (default: false)
• -debugmemlim debug MEMLIMIT (default: false)
• -debugconnpool debug ConnPool (default: false)
• -debugrapidyenc debug rapidyenc (default: false)
• -debugworker debug the download/upload workers (default: false)
• -debugBUG very spammy extra debug lines (default: false)
• -testrapidyenc decode the rapidyenc test files on startup and exit (default: false)
• -logappend append to logfile instead of rotating/overwriting (default: false)
• -logdir set log directory (default: logs)
• -logold rotate log files to .N, 0 disables rotation (default: 0)
• -zzz-shr-testmode only for test compilation on self-hosted runners (default: false)

---

Web interface (-web)

-web starts a small web server instead of processing one NZB: upload NZB files (.nzb, .nzb.gz), queue them, start, stop or cancel jobs, and watch progress and provider statistics in the browser. Jobs run one at a time, oldest first, each with the flags NZBreX was started with (-provider, -cd, -crc32, -checkonly, ...).

Local use, no login (open http://127.0.0.1:11911):

./NZBreX -web -provider provider.json -cd /cache

From other machines a login is required. Create a user once, then start with the same file:

./NZBreX -webpasswd webpasswd -webadduser 'webadmin01|a-long-password'   # or 'auto|auto' to generate both
./NZBreX -web -webaddr ':11911' -webpasswd webpasswd -webssl -webssl-crt fullchain.pem -webssl-key privkey.pem -provider provider.json -cd /cache

• -web [true|false] start the web server (default: false)
• -webaddr listen address (default: 127.0.0.1:11911; :11911 listens on all addresses)
• -webpasswd passwd file with the web users. Without it the server only starts on a loopback address.
• -webadduser 'username|password' (both at least 10 characters) or 'auto|auto': add a user to -webpasswd and exit. A running server picks up new users without a restart.
• -webssl [true|false] serve HTTPS (TLS 1.2+) with -webssl-crt and -webssl-key (default: fullchain.pem, privkey.pem)
• -webhosts host names the server answers to, comma separated, e.g. nzbrex.example.org when it runs behind a reverse proxy. localhost, 127.0.0.1, ::1 and the -webaddr host are always allowed.

After 10 failed logins within 10 minutes, a client address is blocked for 10 minutes. Web users are separate from proxy users. Per-job settings are not supported yet. Details, the JSON API and the security notes are in WEB_JOB_QUEUE.md.

---

NNTP proxy (-proxytcp / -proxytls)

With -proxytcp or -proxytls, NZBreX is a read-only NNTP server: a newsreader or downloader connects to it with a proxy user, and NZBreX fetches each article from the providers in provider.json. Your provider accounts stay on this machine; clients only get proxy users you create, with their own connection limits and expiry dates.

How it answers:

• By message-id (ARTICLE, BODY, HEAD, STAT <id>): asks the providers in provider.json order and returns the first hit. Providers with NoDownload are skipped, and so are the other accounts of a Group once one of them did not have the article. Misses are remembered for a while.
• With -crc32: an article whose yEnc data is damaged is fetched from the next provider instead.
• Group commands (GROUP, LIST, OVER/XOVER, HDR/XHDR, NEXT, LAST, articles by number): only go to providers with "Newsreader": true.
• Articles are passed on unchanged. Posting is not supported.

Start:

./NZBreX -proxyadduser 'reader0001|a-long-password|4|30d|nopost'   # adds a user to .proxypasswd and exits
./NZBreX -provider provider.json -proxytcp 1119                     # plain TCP
./NZBreX -provider provider.json -proxytls 1563 -tlscrt fullchain.pem -tlskey privkey.pem

• -proxytcp port for plain NNTP, e.g. 1119 (default: 0 = off). Listens on all addresses.
• -proxytls port for NNTP over TLS (TLS 1.2+), e.g. 1563 (default: 0 = off). Both can run at once.
• -tlscrt / -tlskey certificate and key for -proxytls (default: fullchain.pem, privkey.pem)
• -proxypasswdfile file with the proxy users (default: .proxypasswd). It is re-read about once a minute, so new users work without a restart.
• -proxyadduser 'username|password|maxconns|expires|nopost': add a user and exit. Username and password need at least 10 characters. maxconns limits the user's connections, and 0 means no limit. expires is a duration (30d, 12h, 45m) or a Unix time; proxy users always expire. 'auto|auto|...' generates username and password and prints them.

Clients have 60 s to log in and are disconnected after 5 minutes without a command. Without users nobody can log in. The proxy can't run in the same process as -web, -nzb or -nzbdir.

---

provider.json options per provider

• Enabled: true|false, enables/disables this provider
• NoDownload: true|false, disables downloads from provider
• NoUpload: true|false, disables upload to provider
• Newsreader: true|false, the account allows group commands (GROUP, LIST, OVER, ...); the NNTP proxy sends those, and articles by number, only to such providers
• Group: string, group providers for deduplication
• Name: string, provider name (for debug/output)
• Host: string, server hostname or IP
• Port: int, server port
• TCPMode: string, tcp|tcp4|tcp6
• PreferIHAVE: bool, prefer IHAVE command
• SSL: bool, use SSL
• SkipSslCheck: bool, ignore certificate errors
• Username: string, account username
• Password: string, account password
• MaxConns: int, max connections
• MaxConnErrors: int, max consecutive fatal connection errors

---

Terminal Output

In -verbose mode, NZBreX prints statistics while running:

• STAT: overall status of STAT command (checked on all providers)
• DONE: items downloaded/uploaded
• SEGM: segment availability (100% = all segments available somewhere)
• DEAD: segments not available on any provider
• DMCA: segments blocked by DMCA (code 451)
• DL: download queue status
• HD: cache status
• UP: reupload queue status
• MEM N/N [C=0|D=0|U=0]: memory slots and running routines

Percentages may be slightly off due to rounding. Rows may appear/disappear depending on matching numbers.

Example output:

2025/05/13 23:48:37  | DONE | [55.309%] (1219/2204) | SEGM [86.706%] (1911) | GETS [55.535%] (1224 / 1911 Q:687) | DISK [55.535%] (1224) | REUP [55.309%] (1219 / 1224 Q:5) | MEM:10/10 [C=5|D=5|U=4]
2025/05/13 23:48:40  |  DL  [ 55%]   SPEED:  5265 KiB/s | (Total: 865.00 MB)
2025/05/13 23:48:40  |  UL  [ 55%]   SPEED:  4838 KiB/s | (Total: 862.00 MB)
...
2025/05/13 23:52:43  |  STAT [100.0%] (2204) | DONE | [93.149%] (2053/2204) | SEGM [95.054%] (2095) | DEAD [4.946%] ( 109) | GETS [93.149%] (2053 / 2204 Q:151) | DISK [93.149%] (2053) | REUP [93.149%] (2053 / 2053 Q:0) | MEM:5/10 [C=0|D=5|U=0]

---

TODOs

• Testing!

---

Ideas and more TODOs

• #0001: watchDir (processor/sessions) done: -nzbdir
• #0010: fix all todos in code!
• #0020: better console output
• #0030: verify uploaded articles
• #0050: config.json does not work
• #0060: streaming (CHECK/TAKETHIS)
• #0080: progressbar (cosmetics)
• ...

---

Troubleshooting

Windows Executable Issues

If you encounter an error like "The code execution cannot proceed because libc++.so.6.dll was not found" on Windows:

1. Download the latest release - This issue has been fixed in recent builds
2. Ensure you're using the correct Windows version - Download the Windows-specific executable from the releases page
3. The executable is self-contained - No additional runtime libraries should be required

The Windows executable is statically linked and only depends on core Windows system libraries that are guaranteed to be available.

Build Issues

• rapidyenc library missing (-tags rapidyenc builds only): run the matching rapidyenc build script first, or build without the tag
• Cross-compilation fails: build without -tags rapidyenc (pure Go cross-compiles anywhere), or install MinGW-w64 and cmake for the C variant
• Link errors: make sure the rapidyenc library matches your target platform

---

Contributing & Reporting Issues

Please open an issue on GitHub Issues or a Discussion if you encounter problems.

Only link NZB files if they are freeware (e.g. debian/ubuntu iso)!

---

• The provider.ygg.json works via yggdrasil network ;)
• Retention of the test servers is short. Articles can expire within minutes. Connections may drop or reply with unexpected codes at any time!
• If the yggdrasil test server does not work, do NOT open an issue! Wait, leave it alone, or run your own!

---

Credits

This software is built using Go (License).

Source based on (github.com/Tensai75/nzbrefresh#commit:cc2a8b7) (MIT License)

This software uses the following external libraries:

• github.com/fatih/color#commit:4c0661 (MIT License)
• github.com/nu11ptr/cmpb (MIT License)
• github.com/Tensai75/nzbparser#commit:a1e0d80 (MIT License)
• github.com/Tensai75/cmpb#commit:16fb79f (MIT License)
• github.com/go-while/go-cpu-mem-profiler (MIT License)
• github.com/go-yenc/yenc (MIT License)

---

Credits to Tensai75!