Make FetchSBOM and FetchVEX stream single files in chunks

[henrymitchellhibbert-lab]

FetchSBOM and FetchVEX were intended to stream large artifacts in chunks, but were not configured correctly to achieve that. Instead they streamed multiple files at a time, but each file was transmitted whole.

This change rewrites these RPCs to each stream one artifact at a time, but the individual artifacts are broken into chunks of bytes.

In response to a request, the server sends the metadata of the file (its identifier and the format of the document) and then streams the file contents in chunks of bytes until the whole file is transmitted.

The server could then look something like this:

func (s *server) FetchSBOM(req *securityservicespb.FetchSBOMRequest, stream grpc.ServerStreamingServer[securityservicespb.FetchSBOMResponse]) error {
	f, err := os.Open(s.sbomPath)
	if err != nil {
		return err
	}
	defer f.Close()
    // Start by sending the metadata
	if err := stream.Send(securityservicespb.FetchSBOMResponse_builder{
		Metadata: securityservicespb.SbomArtifactMetadata_builder{
			Identifier: req.GetIdentifier(),
			Standard:   securityservicespb.SbomDocumentStandard_SBOM_DOCUMENT_STANDARD_CYCLONEDX_JSON,
		}.Build(),
	}.Build()); err != nil {
		return err
	}
    // Send file contents in chunks until EOF
	buf := make([]byte, chunkSize)
	for {
		n, err := f.Read(buf)
		if n > 0 {
			if sendErr := stream.Send(securityservicespb.FetchSBOMResponse_builder{
				Chunk: buf[:n],
			}.Build()); sendErr != nil {
				return sendErr
			}
		}
		if errors.Is(err, io.EOF) {
			return nil
		}
		if err != nil {
			return err
		}
	}
}

[gmacf]

I think we should set both limits on the chunk size and also a recommended chunk size:

• For example, gRPC has a hard limit on 4MB for messages but obviously for streaming performance we want it to be much lower than that.
• Looking at other OpenConfig services, I say we suggest a sensible default of 64KB for chunk sizes (e.g. https://github.com/openconfig/gnoi/blob/e8e54f7bf257d4c7b45556727ef8a6ce75e80d30/file/file.proto#L36)

[gmacf]

What do you think about adding a new field here called size_bytes.

This would let the server tell the client immediately how many bytes to expect in the chunks. If something goes wrong and there's an unexpectedly large (or small) payload, this would catch that . It may also help performance because the client can pre-allocate the memory needed for the document.

[gmacf]

Altough the payload itself is going to be signed (JWS/JSF), parsing that requires putting it through the entire JSON parser. We could get a very cheap verification immediately after the chunks are downloaded by including a sha256 digest (or similar) in this metadata. The client then just does a very quick hash and only proceeds to the JSON parsing if that succeeds.