Repository navigation
Conversation
The output writes log records to a Manticore Search table through the HTTP /bulk endpoint, as NDJSON lines. A record goes as it is: the daemon takes its "id" field as the document id or generates one. Regular mode sends every chunk as one /bulk request of insert or create operations, which Manticore runs as one transaction. A retryable item status, 408, 429 or a 5xx reply without item statuses retries the chunk; any other failure drops it. Shared mode loads a finite source into one table with Manticore's bulk_import. Data requests stage the records, Shared_Indexers sets how many indexers the daemon starts for the load, and one Finish request sent from the exit callback publishes it. The load is published only if every chunk was delivered; otherwise the log names the statement that cancels the staged data. Signed-off-by: Stas Klinov <stas@manticoresearch.com>
The formatter test checks the NDJSON lines of Regular insert and create and of Shared, for records with and without their own id. A bad mode, action, shared_indexers or an empty table fails the start. Signed-off-by: Stas Klinov <stas@manticoresearch.com>
A local receiver answers as the Manticore /bulk endpoint. The Regular case checks the request lines, a retried item and a dropped one. The Shared cases check the requests of one load and the Finish sent at exit, the Finish over TLS with workers 0, an empty source, retried and malformed replies, a failed Finish, and loads that end with pending, exhausted or failed data and so are not published. Signed-off-by: Stas Klinov <stas@manticoresearch.com>
tomatolog
force-pushed
the
feat/out_plugin_manticore
branch
from
October 15, 2026 14:34
a4efcec to
5a12975
Compare
This branch was successfully deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This adds
out_manticore, an output that writes log records to a Manticore Search table through its HTTP/bulkendpoint, as NDJSON. A record goes as it is: the daemon takes itsidfield as the document id or generates one.Regular mode (the default) sends every chunk as one
/bulkrequest ofinsertorcreateoperations, which Manticore runs as one transaction. A retryable item status, 408, 429 or a 5xx reply without item statuses retries the chunk; any other failure drops it.Shared mode loads a finite source, for example a file read with
Exit_On_Eof, into one table with Manticore'sbulk_import. Data requests stage the records,Shared_Indexerssets how many indexers the daemon starts for the load, and one Finish request sent from the exit callback publishes it. The load is published only if every chunk was delivered; otherwise the log names the statement that cancels the staged data. Shared needs a Manticore build with sharedbulk_import(dev packages since 29.15.0).Options:
table(required),mode(RegularorShared),action(insertorcreate, Regular),shared_indexers(Shared),buffer_size,http_user,http_passwd, and the usual network, TLS andworkersoptions.Tests:
tests/runtime/out_manticore.c: the formatter output of Regularinsertandcreateand of Shared, for records with and without an id, and settings that fail the start;tests/integration/scenarios/out_manticore: a local receiver in place of/bulkchecks the Regular requests, a retried and a dropped item; the requests of a Shared load and its Finish at exit, the Finish over TLS withworkers 0, retried and malformed replies, a failed Finish, and loads that end with pending, exhausted or failed data and so are not published.Both pass locally; the integration scenario also under
VALGRIND=1 VALGRIND_STRICT=1.Enter
[N/A]in the box, if an item is not applicable to your change.Testing
Before we can approve your change; please submit the following in a comment:
If this is a change to packaging of containers or native binaries then please confirm it works for all targets.
ok-package-testlabel to test for all targets (requires maintainer to do).Documentation
Doc PR: to follow
Backporting
Example configuration
Manticore Search 29.15.0 dev (
manticoresearch/manticore:dev-29.15.0-13685b0, the first build with sharedbulk_import) runs as the containerflb-pr-manticoreon the Docker network of Fluent Bit, with the tables created first:/out/records.jsonholds 1000 lines like{"id": 1, "message": "record 1 from Fluent Bit", "status": 201}, and/out/parsers.confdefines ajsonparser.Regular,
fluent-bit-regular.yaml:Shared,
fluent-bit-shared.yaml, differs in two lines:Both runs exit 0 and leave 1000 rows in their table (
SELECT COUNT(*)).Debug log output
Regular
Shared: two data flushes, then the Finish after the workers stop
Valgrind output
Fluent Bit built with
-DFLB_DEV=On -DFLB_VALGRIND=On -DFLB_JEMALLOC=Off, run asvalgrind --leak-check=full --show-leak-kinds=definite,indirect,possible --error-exitcode=99 ./bin/fluent-bit -c <config>:Regular
Shared
flb-rt-out_manticore