Race condition with overwrite=true on proxmox_virtual_environment_file -> Error: failed to read file from

[MacherelR]

Summary

I’m seeing intermittent errors with proxmox_virtual_environment_file when using overwrite = true for ISO uploads.

In some runs, the resource logs:

Warning: the existing file "secondary_01:iso/<name>_seed.iso" has been overwritten by the resource
Error: failed to read file from "secondary_01:iso/<name>_seed.iso"

The ISO file does exist in Proxmox afterwards, so it looks like the upload actually succeeds but the provider fails when reading back metadata. This feels like a timing / race condition between the upload and the follow-up read.
I feel like this happens because of parallelism as I create multiple ISOs and for the one getting the error I do not have the classic
module.cloud_init["service"].proxmox_virtual_environment_file.seed_iso: Creation complete after Xs message.

Basically an example of logs I get would be:

module.cloud_init["ex-01-svc-2"].proxmox_virtual_environment_file.seed_iso: Creating...
module.cloud_init["ex-01-svc-3"].proxmox_virtual_environment_file.seed_iso: Creating...
module.cloud_init["ex-01-svc-4"].proxmox_virtual_environment_file.seed_iso: Creating...
module.cloud_init["ex-01-svc-1"].proxmox_virtual_environment_file.seed_iso: Creating...
module.cloud_init["ex-01-svc-0"].proxmox_virtual_environment_file.seed_iso: Creating...
module.cloud_init["ex-01-svc-2"].proxmox_virtual_environment_file.seed_iso: Creation complete after 3s [id=secondary_01:iso/ex-01-svc-2_a6c50c31a1fa1bdb915999bca57fa8c1_seed.iso]
module.cloud_init["ex-01-svc-1"].proxmox_virtual_environment_file.seed_iso: Creation complete after 5s [id=secondary_01:iso/ex-01-svc-1_ff87053e8780489c95c9a793fcd16b52_seed.iso]
module.cloud_init["ex-01-svc-0"].proxmox_virtual_environment_file.seed_iso: Creation complete after 5s [id=secondary_01:iso/ex-01-svc-0_63195b7bc9c641aa6b700928a2a9655d_seed.iso]
module.cloud_init["ex-01-svc-4"].proxmox_virtual_environment_file.seed_iso: Creation complete after 5s [id=secondary_01:iso/ex-01-svc-4_86ade32ca8fff9f782e0250e8e15299c_seed.iso]
╷
│ Error: failed to read file from "secondary_01:iso/ex-01-svc-3_eb996d813bfdf3150a0f3b20d570b0bf_seed.iso"
│ 
│   with module.cloud_init["ex-01-svc-3"].proxmox_virtual_environment_file.seed_iso,
│   on ../modules/cloud_init/cloud_init.tf line 87, in resource "proxmox_virtual_environment_file" "seed_iso":
│   87: resource "proxmox_virtual_environment_file" "seed_iso" {
│ 
╵

With -parallelism=1, the problem is much harder to reproduce (I haven’t hit the error again yet), but I still see the overwrite warning, so the same code path is in use.

I’d like to know if this is a known limitation, or if there’s a recommended pattern for this use case (cloud-init ISO with overwrite = true).

Environment

CLI: OpenTofu (via atmos terraform deploy)

Atmos: atmos terraform deploy -s

Terraform/OpenTofu flags: sometimes -parallelism=10 (default), sometimes -parallelism=1

Use case: generate cloud-init seed ISOs locally (user-data / meta-data / network-config via genisoimage), then upload them to Proxmox and attach to VMs to avoid using ssh for cloud-init.

[bpg-dev]

Are you writing are file to a shared storage? The same file simultaneously from multiple nodes?
Are you sure it is a valid use case?

[MacherelR]

It is indeed a shared storage through 3 nodes (ceph) but at the moment of the error, the terraform is the only user to use it.

[bpg-dev]

Thanks for confirming, @MacherelR. That explains the issue.

This is a race condition caused by Ceph's eventual consistency. Here's what happens:

1. Provider uploads the file and waits for the Proxmox upload task to complete — this succeeds
2. Provider immediately reads back the file metadata via a directory listing (GET /storage/{storage}/content)
3. On Ceph, the file physically exists but isn't yet visible in the directory listing due to metadata propagation delay
4. Provider sees "not found" and reports the error

With -parallelism=1 you don't hit it because each upload has time to propagate before the next one starts. With parallel uploads, multiple simultaneous writes create metadata contention on Ceph, widening the consistency window.

The fix is straightforward: retry the read-back with backoff when the file isn't found after a successful upload, instead of failing immediately. I'll create an issue to track this.

As a workaround until the fix is released, you can use -parallelism=1 or add depends_on between the file resources to serialize the uploads.

[bpg-dev]

was fixed in #2571