From de2299df80cc58e4fef8ee205b844d2d3eaef2e1 Mon Sep 17 00:00:00 2001 From: Adam Wright Date: Mon, 21 Sep 2026 21:05:11 +0000 Subject: [PATCH 1/2] Budget the diagram exporters, on the paths production already has CURATOR-REPORT.md listed rate limiting in front of the render service as the outstanding condition before it fronts reactome.org. Doing that turned up something worth more: the Java exporters have no budget either, and they are the ones with a history. `/ContentService/exporter/*` is where crawlers exhausted Tomcat's heap and took the origin down, and today it is covered only by the site-wide 100r/s, which is sized for page assets rather than for a process drawing a diagram. So the budget goes on both, and they share a zone. It belongs to the work -- drawing a diagram -- rather than to which implementation serves it, so a caller cannot spend it twice by alternating between the Java exporter and the node renderer. 2r/s with a burst of 8, which is the node service's own queue depth (RENDER_CONCURRENCY=2, RENDER_QUEUE=8); past that it is rejecting anyway, so a deeper burst only buys a larger pile of work to throw away. Putting it on the ContentService path is also the point rather than a coincidence. Those URLs exist in production today and are what #273 moves to node, under the same names -- so the budget is already where it needs to be when the implementation changes underneath it, and `/RenderService` stops being the only protected way in. Verified rather than reasoned, against the running services: the exporter still returns a PNG through the new block, an ordinary /ContentService/ path is unaffected, the node renderer still returns a 11952x6700 PNG and an SVG, and both budgets refuse past the burst -- 8 through then 503. Co-Authored-By: Claude Opus 5 --- CURATOR-REPORT.md | 10 +++++++-- deploy/nginx/common/routes.conf | 40 +++++++++++++++++++++++++++++++++ deploy/nginx/dev.conf | 26 +++++++++++++++++++++ deploy/nginx/production.conf | 26 +++++++++++++++++++++ deploy/nginx/release.conf | 26 +++++++++++++++++++++ 5 files changed, 126 insertions(+), 2 deletions(-) diff --git a/CURATOR-REPORT.md b/CURATOR-REPORT.md index 7b84e229..a7f90dad 100644 --- a/CURATOR-REPORT.md +++ b/CURATOR-REPORT.md @@ -193,8 +193,14 @@ Not bugs, so as not to waste your time: ## Waiting on someone else - **The render service runs in a container now** (`restart: unless-stopped`), so a - reboot no longer stops GIF and PPTX. Still outstanding before this fronts - reactome.org: rate limiting in front of it. + reboot no longer stops GIF and PPTX. **Rate limiting is now in front of it**, so + the condition this listed is met: nginx gives `/RenderService/` its own budget + of 2 requests a second with a burst of 8 — the service's own queue depth, past + which it is rejecting anyway. The site-wide limit it previously fell under is + 100 a second, which is sized for page assets, not for a headless browser + drawing a 12000-pixel canvas. Crawlers on the old + `/ContentService/exporter/*` URLs are what exhausted Tomcat's heap and took the + origin down; this is the bound that stops that repeating. - **Cloudflare cache purge** — one-off, for figures cached before 2026-08-20. Nothing new is cached now. - **[#139](https://github.com/reactome/WebsiteAngular/issues/139) native cytoscape diff --git a/deploy/nginx/common/routes.conf b/deploy/nginx/common/routes.conf index 0d339f5b..0c589371 100644 --- a/deploy/nginx/common/routes.conf +++ b/deploy/nginx/common/routes.conf @@ -64,6 +64,27 @@ location ~ ^/ContentService/data/person/[^/]+/(authored|reviewed)(Pathways|React proxy_pass http://content_node; } +# The diagram exporters, which are the expensive part of ContentService and the +# only part with a history of taking the origin down: crawlers on these URLs +# exhausted Tomcat's heap. They have had no budget of their own until now -- +# only the site-wide 100r/s, which is sized for page assets. +# +# Longest-prefix wins in nginx, so this claims the exporter paths and everything +# else still falls to /ContentService/ below, in whichever order they appear. +# +# Shares `render_rate` with /RenderService/ deliberately. The budget belongs to +# the work -- drawing a diagram -- rather than to which implementation happens +# to serve it, so a caller cannot spend it twice by alternating between the Java +# exporter and the node renderer. That also means it is already on the right +# path for #273, which moves these exports to node under these same URLs. +location /ContentService/exporter/ { + limit_req zone=render_rate burst=8 nodelay; + include /etc/nginx/common/upstream-proxy.conf; + proxy_pass http://content/ContentService/exporter/; + proxy_read_timeout 300s; + proxy_send_timeout 300s; +} + location /ContentService/ { include /etc/nginx/common/upstream-proxy.conf; proxy_pass http://content/ContentService/; @@ -186,6 +207,25 @@ location /mcp { proxy_read_timeout 300s; } +# The render service. Routed here rather than left to fall through to the site +# server -- which proxies it via proxy.conf.js and would still work -- because +# a budget can only be applied where the path is named. +# +# `/RenderService/` with a trailing slash on both sides, so the prefix is +# stripped: the service serves `/render/{id}.{ext}` at its root. That matches +# what proxy.conf.js already does with `pathRewrite`. +location /RenderService/ { + limit_req zone=render_rate burst=8 nodelay; + include /etc/nginx/common/upstream-proxy.conf; + proxy_pass http://render/; + # A large diagram takes tens of seconds and returns megabytes. The site's + # 300s matches what the exporters needed; buffering off so a slow render + # streams rather than being held whole in nginx first. + proxy_buffering off; + proxy_read_timeout 300s; + proxy_send_timeout 300s; +} + # The site itself. location / { include /etc/nginx/common/upstream-proxy.conf; diff --git a/deploy/nginx/dev.conf b/deploy/nginx/dev.conf index b8d57c03..a2aea2e3 100644 --- a/deploy/nginx/dev.conf +++ b/deploy/nginx/dev.conf @@ -93,6 +93,18 @@ upstream mcp { keepalive 16; } +# The headless render service, which draws diagram figures (PNG, SVG, GIF, +# PPTX, PDF) with the site's own renderer. +# +# Reached today through the site server, which proxies /RenderService from +# proxy.conf.js. Named here so nginx can put a budget in front of it before it +# ever fronts reactome.org -- which CURATOR-REPORT.md lists as the outstanding +# condition on that. +upstream render { + server 127.0.0.1:4310; + keepalive 16; +} + include /etc/nginx/common/websocket.conf; include /etc/nginx/common/cloudflare-real-ip.conf; include /etc/nginx/common/block-ai-crawlers.conf; @@ -113,6 +125,20 @@ limit_req_zone $binary_remote_addr zone=dev_rate:10m rate=100r/s; # page assets and would let one client queue analyses as fast as it can ask. limit_req_zone $binary_remote_addr zone=mcp_rate:10m rate=2r/s; +# A budget for renders, two orders of magnitude tighter than the site's. +# +# A render is a headless browser drawing a canvas -- R-HSA-109606 comes back as +# a 7 MB, 11952x6700 PNG -- so the cost per request is seconds of CPU rather +# than a file read. The site's 100r/s is sized for page assets, and crawlers on +# the old /ContentService/exporter/* URLs are what exhausted Tomcat's heap and +# took the origin down. That is the failure this exists to prevent repeating. +# +# 2r/s with a burst of 8, which is the service's own queue depth +# (RENDER_CONCURRENCY=2, RENDER_QUEUE=8): past that it is rejecting anyway, so +# a burst larger than the queue only buys a deeper pile of work to throw away. +limit_req_zone $binary_remote_addr zone=render_rate:10m rate=2r/s; + + limit_conn_zone $binary_remote_addr zone=dev_conn:10m; # The retired hostnames stay retired, and so does anything else pointed here. diff --git a/deploy/nginx/production.conf b/deploy/nginx/production.conf index 527cf3fd..42225b36 100644 --- a/deploy/nginx/production.conf +++ b/deploy/nginx/production.conf @@ -102,6 +102,18 @@ upstream mcp { keepalive 16; } +# The headless render service, which draws diagram figures (PNG, SVG, GIF, +# PPTX, PDF) with the site's own renderer. +# +# Reached today through the site server, which proxies /RenderService from +# proxy.conf.js. Named here so nginx can put a budget in front of it before it +# ever fronts reactome.org -- which CURATOR-REPORT.md lists as the outstanding +# condition on that. +upstream render { + server 127.0.0.1:4310; + keepalive 16; +} + include /etc/nginx/common/websocket.conf; include /etc/nginx/common/cloudflare-real-ip.conf; include /etc/nginx/common/block-ai-crawlers.conf; @@ -119,6 +131,20 @@ limit_req_zone $binary_remote_addr zone=prod_rate:20m rate=600r/s; # page assets and would let one client queue analyses as fast as it can ask. limit_req_zone $binary_remote_addr zone=mcp_rate:10m rate=2r/s; +# A budget for renders, two orders of magnitude tighter than the site's. +# +# A render is a headless browser drawing a canvas -- R-HSA-109606 comes back as +# a 7 MB, 11952x6700 PNG -- so the cost per request is seconds of CPU rather +# than a file read. The site's 100r/s is sized for page assets, and crawlers on +# the old /ContentService/exporter/* URLs are what exhausted Tomcat's heap and +# took the origin down. That is the failure this exists to prevent repeating. +# +# 2r/s with a burst of 8, which is the service's own queue depth +# (RENDER_CONCURRENCY=2, RENDER_QUEUE=8): past that it is rejecting anyway, so +# a burst larger than the queue only buys a deeper pile of work to throw away. +limit_req_zone $binary_remote_addr zone=render_rate:10m rate=2r/s; + + limit_conn_zone $binary_remote_addr zone=prod_conn:20m; server { diff --git a/deploy/nginx/release.conf b/deploy/nginx/release.conf index 05b7190a..8644d10b 100644 --- a/deploy/nginx/release.conf +++ b/deploy/nginx/release.conf @@ -96,6 +96,18 @@ upstream mcp { keepalive 16; } +# The headless render service, which draws diagram figures (PNG, SVG, GIF, +# PPTX, PDF) with the site's own renderer. +# +# Reached today through the site server, which proxies /RenderService from +# proxy.conf.js. Named here so nginx can put a budget in front of it before it +# ever fronts reactome.org -- which CURATOR-REPORT.md lists as the outstanding +# condition on that. +upstream render { + server 127.0.0.1:4310; + keepalive 16; +} + include /etc/nginx/common/websocket.conf; include /etc/nginx/common/cloudflare-real-ip.conf; include /etc/nginx/common/block-ai-crawlers.conf; @@ -112,6 +124,20 @@ limit_req_zone $binary_remote_addr zone=rel_rate:10m rate=100r/s; # page assets and would let one client queue analyses as fast as it can ask. limit_req_zone $binary_remote_addr zone=mcp_rate:10m rate=2r/s; +# A budget for renders, two orders of magnitude tighter than the site's. +# +# A render is a headless browser drawing a canvas -- R-HSA-109606 comes back as +# a 7 MB, 11952x6700 PNG -- so the cost per request is seconds of CPU rather +# than a file read. The site's 100r/s is sized for page assets, and crawlers on +# the old /ContentService/exporter/* URLs are what exhausted Tomcat's heap and +# took the origin down. That is the failure this exists to prevent repeating. +# +# 2r/s with a burst of 8, which is the service's own queue depth +# (RENDER_CONCURRENCY=2, RENDER_QUEUE=8): past that it is rejecting anyway, so +# a burst larger than the queue only buys a deeper pile of work to throw away. +limit_req_zone $binary_remote_addr zone=render_rate:10m rate=2r/s; + + limit_conn_zone $binary_remote_addr zone=rel_conn:10m; server { From 41e8e8a0992b93ea07f2af5b259c13743b691a7e Mon Sep 17 00:00:00 2001 From: Adam Wright Date: Mon, 21 Sep 2026 21:12:33 +0000 Subject: [PATCH 2/2] Budget what draws, not everything under /exporter/ The prefix version of this would have throttled reaction pages to protect something they do not use. `/ContentService/exporter/` is not one kind of thing: `/exporter/reaction/{id}/diagram` is a 2.9 KB JSON document served in 0.17s and it is on the critical path for every reaction page, while a render of R-HSA-109606 on the same prefix is 454 KB, and 7 MB at full quality. Measured both before changing it, which is the only reason the difference was visible: I had written the prefix, tested that it refused past the burst, and that test passed exactly as happily on the endpoint that must not be refused. A regex location now names the four that draw -- diagram, fireworks, document, event -- so it is tried before the /ContentService/ prefix and everything else, the reaction JSON included, falls through unbudgeted. Verified: fifteen rapid reaction-diagram requests all 200, the render path refuses past the burst, and a real render returns its 454 KB PNG once the budget refills. That last check matters because the run before it returned 503 -- my own flood had spent the budget -- and stopping there would have shipped on a failure I had caused. Co-Authored-By: Claude Opus 5 --- CURATOR-REPORT.md | 5 ++++- deploy/nginx/common/routes.conf | 25 +++++++++++++++++-------- 2 files changed, 21 insertions(+), 9 deletions(-) diff --git a/CURATOR-REPORT.md b/CURATOR-REPORT.md index a7f90dad..3573d12b 100644 --- a/CURATOR-REPORT.md +++ b/CURATOR-REPORT.md @@ -196,7 +196,10 @@ Not bugs, so as not to waste your time: reboot no longer stops GIF and PPTX. **Rate limiting is now in front of it**, so the condition this listed is met: nginx gives `/RenderService/` its own budget of 2 requests a second with a burst of 8 — the service's own queue depth, past - which it is rejecting anyway. The site-wide limit it previously fell under is + which it is rejecting anyway. The same budget now covers the Java exporters + that draw (`/ContentService/exporter/diagram|fireworks|document|event`), which + had none; the reaction-diagram JSON on the same prefix is deliberately outside + it, being 2.9 KB in 0.17s and on every reaction page's critical path. The site-wide limit it previously fell under is 100 a second, which is sized for page assets, not for a headless browser drawing a 12000-pixel canvas. Crawlers on the old `/ContentService/exporter/*` URLs are what exhausted Tomcat's heap and took the diff --git a/deploy/nginx/common/routes.conf b/deploy/nginx/common/routes.conf index 0c589371..acca254d 100644 --- a/deploy/nginx/common/routes.conf +++ b/deploy/nginx/common/routes.conf @@ -64,23 +64,32 @@ location ~ ^/ContentService/data/person/[^/]+/(authored|reviewed)(Pathways|React proxy_pass http://content_node; } -# The diagram exporters, which are the expensive part of ContentService and the -# only part with a history of taking the origin down: crawlers on these URLs -# exhausted Tomcat's heap. They have had no budget of their own until now -- -# only the site-wide 100r/s, which is sized for page assets. +# The exporters that actually draw something, which are the expensive part of +# ContentService and the only part with a history of taking the origin down: +# crawlers on these URLs exhausted Tomcat's heap. They have had no budget of +# their own until now -- only the site-wide 100r/s, which is sized for page +# assets. # -# Longest-prefix wins in nginx, so this claims the exporter paths and everything -# else still falls to /ContentService/ below, in whichever order they appear. +# Named individually rather than as `/exporter/`, because that prefix is not all +# one thing. `/exporter/reaction/{id}/diagram` is a 2.9 KB JSON document served +# in 0.17s and it is on the critical path for every reaction page; a render of +# R-HSA-109606 is 454 KB in 0.55s, and at full quality 7 MB. Measured, after +# writing the prefix version and realising it would throttle reaction pages to +# protect something they do not use. +# +# A regex location, so it is tried before the `/ContentService/` prefix below and +# anything not listed here -- the reaction JSON included -- falls through to it +# unbudgeted. # # Shares `render_rate` with /RenderService/ deliberately. The budget belongs to # the work -- drawing a diagram -- rather than to which implementation happens # to serve it, so a caller cannot spend it twice by alternating between the Java # exporter and the node renderer. That also means it is already on the right # path for #273, which moves these exports to node under these same URLs. -location /ContentService/exporter/ { +location ~ ^/ContentService/exporter/(diagram|fireworks|document|event)/ { limit_req zone=render_rate burst=8 nodelay; include /etc/nginx/common/upstream-proxy.conf; - proxy_pass http://content/ContentService/exporter/; + proxy_pass http://content; proxy_read_timeout 300s; proxy_send_timeout 300s; }