Serve the diagram and reaction figure exports from node, on the ContentService paths

[adamjohnwright]

The node render service (tools/render) already produces every diagram figure the site offers, drawn by the site's own renderer rather than a second reimplementation of it. It answers on /RenderService/render/<id>.<ext>, a path that exists on beta and not on production.

That extra path is the problem. The figures it makes are the ones /ContentService/exporter/diagram/<id>.<ext> and /exporter/reaction/<id>.<ext> have always made, so a second URL for the same thing means external consumers keep the old one, our own tests cannot assert against a path that exists everywhere, and the two can drift.

Proposal: node answers the existing ContentService paths, with Tomcat behind it as a backup upstream — the arrangement already proven for /data/content/ in #269. Nothing outside has to change, and anything node does not answer still reaches Java.

What already lines up

/render/:name.:ext is the same request in different clothes: token → token, sel → select, view → view, plus our own subpathways, dark and scale. The routing is an nginx location for the prefix pointing at the node upstream.

What has to be closed first

jpeg and jpg. Java's enum is png, jpg, jpeg, svg, gif; the render service is svg, png, pdf, gif, pptx and returns 400 for jpeg. Taking the path over without it is a format regression. Small: the canvas is already there, and ehld.service.ts already handles JPEG having no alpha channel.

Eleven parameters we do not implement: quality, flg, flgInteractors, title, margin, ehld, diagramProfile, analysisProfile, resource, expColumn, includeRef. Our own site uses almost none of them, but this is a public URL and we cannot enumerate who calls it. Each needs a decision — honour it, or reject the request naming the parameter. Ignoring one silently is the outcome to avoid: flg dropped on the floor renders a figure without the flagged entity and returns 200, which no caller can detect.

Rate limiting in front, before the switch, not after. A render costs seconds where a static file costs milliseconds, and this path is public and crawled. That work is already on the list before the render service can front reactome.org; this issue depends on it rather than repeating it.

Why it is likely to be an improvement rather than a risk

The render service answers 503 beyond a bounded queue instead of accumulating work, caches on the request, and coalesces concurrent requests for the same pathway into one render — four concurrent became one, measured. The exports it replaces have none of that.

What it unblocks

Five tests in e2e/downloads.spec.ts assert on figure formats and are gated on the render service being up. CI does not run it, so they never execute there; where it is running they fail for want of a recording. Once the figures come from a path that exists on every deployment, those assertions can live in the release suite and run against real output.

Order

1. add jpeg/jpg
2. decide each unimplemented parameter: honour, or reject naming it
3. rate limiting in front
4. route the two paths to node with Tomcat as backup
5. move the figure-format assertions to the release suite

[adamjohnwright]

Read the Java (ImageExporterController.diagramImage). It changes the shape of this: three of the things listed are not optional flags at all, and the eleven parameters split in two ways that matter.

Nothing is to be reimplemented until we know who uses what. Adam is scanning production's access logs. Recorded here so the scope is unambiguous when that comes back.

Not a parameter: the endpoint resolves the identifier

diagramService.getDiagramResult(identifier) maps an id to its containing diagram and returns the events to pre-select, so asking for a subpathway gives the parent's diagram with that subpathway highlighted, and asking for a reaction gives the diagram it lives in with the reaction selected. sel is appended to that list, not a replacement for it:

List<String> toSelect = result.getEvents();
if (sel != null) toSelect.addAll(sel);

Measured against production:

R-HSA-5205685 (a subpathway with no diagram of its own)   java 200, 2243x1053
                                                          node 500

Node has no equivalent. Taking the path over today turns every subpathway and reaction URL into a 500 — far more consumers than any of the flags.

Defaulted vs opt-in, which decides what the logs can tell us

Opt-in — no default, so they only matter if someone passes them, and the log scan settles it: flg, sel, expColumn, token.

Defaulted — in force on every request whether or not anyone names them: quality=5, title=true, margin=15, ehld=true, flgInteractors=true, diagramProfile=Modern, analysisProfile=Standard, resource=TOTAL. No one has to appear in a log for these to be shaping every image production serves. Not honouring them is not "dropping an unused parameter", it is changing every existing caller's output.

Two of them are not subtle. Measured on production for R-HSA-109606:

quality=1    302x179      19KB
quality=5   3016x1787    444KB   <- the default
quality=10  9048x5360   2249KB

title=false 3016x1702            <- default true adds an 85px caption strip

Our renders draw no title at all and size themselves differently, so a switch without these silently changes the picture every existing caller receives.

One behaviour not to reproduce

try { args.setFlags(searchManager.getDiagramFlagging(result.getDiagramStId(), flg, flgInteractors)); }
catch (SolrSearcherException e) {
    //Nothing to be flagged
}

A flag search that fails returns the diagram unflagged, with a 200. The caller asked for something to be marked and cannot tell that it was not. Worth fixing when we reimplement rather than copying.

Revised order

1. Wait for the log scan — who calls these paths, with which parameters, from which referrers. External embedders of the old pathway browser are the consumers we cannot otherwise see.
2. Subpathway/reaction resolution, which is prerequisite to everything and independent of the scan
3. jpeg/jpg
4. The defaulted parameters, since they apply whether or not anyone asks
5. The opt-in ones, as the scan directs
6. Rate limiting in front
7. Route the paths to node with Tomcat as backup

[adamjohnwright]

The resolution behaviour is now precisely known: the Cypher is in graph-core's compiled DiagramRepository, and it runs correctly against our graph.

Three cases, one query, keyed on stId (there is a dbId twin):

MATCH (d:Pathway{stId:$stId, hasDiagram:true})
RETURN d.stId AS diagramStId, [] AS events, d.diagramWidth AS width, d.diagramHeight AS height, 1 AS level
UNION
MATCH path=(d:Pathway{hasDiagram:true})-[:hasEvent*1..100]->(s:Pathway{stId:$stId, hasDiagram:false})
WHERE single(x IN nodes(path) WHERE (x:Pathway) AND x.hasDiagram)
OPTIONAL MATCH aux=(s)-[:hasEvent*1..100]->(rle:ReactionLikeEvent)
WHERE none(x IN nodes(aux) WHERE (x:Pathway) AND x.hasDiagram)
WITH DISTINCT d, s, collect(DISTINCT rle.stId) AS events
OPTIONAL MATCH depth=shortestPath((tlp:TopLevelPathway)-[:hasEvent*1..100]->(d))
WHERE NOT (d:TopLevelPathway)
RETURN d.stId AS diagramStId, events, d.diagramWidth AS width, d.diagramHeight AS height, size(nodes(depth)) AS level
ORDER BY level LIMIT 1
UNION
MATCH path=(d:Pathway{hasDiagram:true})-[:hasEvent*1..100]->(r:ReactionLikeEvent{stId:$stId})
WHERE single(x IN nodes(path) WHERE (x:Pathway) AND x.hasDiagram)
WITH DISTINCT d, r
OPTIONAL MATCH depth=shortestPath((tlp:TopLevelPathway)-[:hasEvent*1..100]->(d))
WHERE NOT (d:TopLevelPathway)
RETURN d.stId AS diagramStId, [r.stId] AS events, d.diagramWidth AS width, d.diagramHeight AS height, size(nodes(depth)) AS level
ORDER BY level LIMIT 1

Run against our graph:

R-HSA-109606  (has a diagram)  -> itself,          0 events   3107x1718    81ms
R-HSA-5205685 (a subpathway)   -> R-HSA-5205647,  16 events   2225x968    928ms
R-HSA-6805479 (a reaction)     -> R-HSA-6804756,   1 event    2412x1584  1463ms
R-HSA-1640170 (top-level EHLD) -> itself,          0 events   651x136       3ms

Two things fall out of it that were not obvious.

The selection is a list, not a value. Sixteen events for one subpathway. /render's select takes a single string and hands it to the diagram's own lookup, so it needs to accept several before this can be wired up.

width and height come back too, which is what Java's result.getSize() feeds — the raster semaphore that bounds concurrent pixel work. Worth keeping in mind when this endpoint inherits the traffic that once took the origin down: knowing a render's size before drawing it is how you refuse the expensive ones.

And it is slow enough to matter: ~1s for a subpathway, ~1.5s for a reaction, against 3–80ms when the id already has a diagram. Cacheable — it changes only with the release — but not free.

Where it should live, which is a decision rather than a detail

The render service has no graph access by design: it talks to the site over HTTP and drives a browser. Adding Neo4j credentials to a container that runs Chromium widens what a browser compromise reaches.

The alternative reads better. content-node owns the ContentService URL and does the resolution; the render service draws what it is told. content-node already has the graph and already answers /ContentService/data/… paths with Tomcat behind it, so /ContentService/exporter/diagram/<id>.<ext> is the same arrangement one path along: it validates the parameters, resolves the id to a diagram plus a selection, and asks the render service for that picture. The render service stays graph-free and keeps one job.

That also puts the parameter work where it belongs — the eleven parameters are a ContentService contract, not a renderer's.

Recommendation, but not a decision I should make alone: it changes what both services are for.

[adamjohnwright]

Decided

From outside there are two services, ContentService and AnalysisService, exactly as the Java deployment presents them. A reader or an external consumer should never meet a /RenderService URL for something that has always been a ContentService endpoint. Seamless is the requirement; which process answers is our business, not theirs.

That decides a lot:

• /ContentService/exporter/diagram/<id>.<ext> and /exporter/reaction/<id>.<ext> are answered by content-node, with Tomcat behind as a backup upstream — the arrangement already proven for /data/content/ in Make the node content service live: compose, nginx from the repo, and a fallback to Tomcat #269.
• content-node validates the parameters, resolves the identifier to a diagram plus its selection, and applies the caching rules. It already holds a read-only graph connection, so the resolution query costs no new access.
• When a request needs a picture drawn, content-node asks the render service over loopback and returns the bytes.
• /RenderService becomes internal. It stays loopback-only and stops being something anyone outside has to know about. The app's own RENDER_SERVICE calls move to ContentService paths.

Why not one process

The render service is a browser, and the numbers argue for keeping the seam:

content-node   1.16GB image    53MB resident
render         2.16GB image   229MB resident, idle

Merging would put the contents, DOI and contributors endpoints — which answer in ~4ms from memory and which the site depends on — in the same process as Chromium holding a large canvas. The compose file already notes that the render container exists partly so that restarting it is a safe response, there being no state to lose but a cache on a volume. That stops being true once the content caches share the process.

So: one service boundary, two processes. Outside sees one ContentService; inside, the browser keeps its own container where it can be restarted, memory-bounded and rate-limited on its own terms.

A consequence worth noting

The render service does not need Neo4j after all. Making it read-only would have made that safe, but with resolution in content-node it is not needed — one graph connection, in the service that already has one.

Order

1. content-node gains the resolution query (graph-core's, verified: 3 cases, ~1s for a subpathway, cacheable per release)
2. content-node answers /exporter/diagram/<id>.<ext>, delegating the drawing over loopback
3. the render page accepts a list of selections — 16 events for one subpathway, and select currently takes a single string
4. the defaulted parameters: quality, title, margin, ehld, the two profiles, resource, flgInteractors
5. flg, sel, expColumn as the production log scan directs
6. rate limiting in front, then route the paths over
7. the app's own RENDER_SERVICE calls move to ContentService paths, and /RenderService stops being public

[adamjohnwright]

Adversarial review of this issue's own analysis

I had written that the diagram exporter has eleven parameters we do not implement, and listed includeRef among them. That was wrong, and chasing it turned up something that matters more than the miscount.

view and includeRef are not exporter parameters at all. A Swagger customiser adds them to every operation in the service:

public OpenApiCustomiser addGlobalQueryParameters(Parameter viewParameter, Parameter includeRefParameter) {
    return openApi -> openApi.getPaths().values().forEach(path ->
            path.readOperations().forEach(operation ->
                    operation.addParametersItem(viewParameter).addParametersItem(includeRefParameter)));
}

view is flatten | nested | nested-aggregated, default flatten; includeRef replaces element refs with JSOG {@ref} objects. Both shape the JSON serialisation, service-wide.

They are real, and they change responses. Measured against Java:

/data/query/R-HSA-109581                        13500 bytes
/data/query/R-HSA-109581?includeRef=true        13940
/data/query/R-HSA-109581?view=nested-aggregated 14233

The endpoints node already serves are not affected, which I checked rather than assumed:

/data/content/toc                        249794 bytes
/data/content/toc?includeRef=true        249794
/data/content/toc?view=nested-aggregated 249794

Those three are built by ContentPageManager rather than the graph serialiser, so the global parameters pass through them without effect — node ignoring them matches Java ignoring them. No gap today.

But it is a trap for everything after. Any endpoint node takes over that returns graph objects — /data/query/{id} being the obvious next one — is subject to both, and would silently ignore them. They will not appear in that endpoint's own signature, because nothing declares them there; they are attached globally at documentation time and handled by a message converter. Reading the controller, which is what I did here, does not show them.

And a name collision to resolve before the takeover. Our render endpoint already has a view parameter meaning reaction. ContentService's global view means flatten | nested | nested-aggregated. The moment these paths move under /ContentService/, one name has two meanings. Ours should be renamed rather than overloaded — layout=reaction or similar — since theirs is service-wide and predates us.

Corrected parameter list

Own to the diagram exporter, none implemented: quality, flg, flgInteractors, sel, title, margin, ehld, diagramProfile, analysisProfile, resource, expColumn — eleven, with token the twelfth and the only one we do implement. The reaction exporter has the same set minus ehld.

Service-wide, and applying to every ported endpoint: view, includeRef.