Kubo doesn't throw a proper error if I add content to the network with offline/inaccessible http routers

[Rinse12]

Checklist

• This is a bug report, not a question. Ask questions on discuss.ipfs.tech.
• I have searched on the issue tracker for my bug.
• I am running the latest kubo version or have an issue updating.

Installation method

dist.ipfs.tech or ipfs-update

Version

Kubo version: 0.37.0
Repo version: 17
System version: amd64/linux
Golang version: go1.25.0

Config

{
  "API": {
    "HTTPHeaders": {}
  },
  "Addresses": {
    "API": "/ip4/127.0.0.1/tcp/54322",
    "Announce": [],
    "AppendAnnounce": [],
    "Gateway": "/ip4/127.0.0.1/tcp/54323",
    "NoAnnounce": [],
    "Swarm": [
      "/ip4/0.0.0.0/tcp/54321",
      "/ip6/::/tcp/54321",
      "/ip4/0.0.0.0/udp/54321/webrtc-direct",
      "/ip4/0.0.0.0/udp/54321/quic-v1",
      "/ip4/0.0.0.0/udp/54321/quic-v1/webtransport",
      "/ip6/::/udp/54321/webrtc-direct",
      "/ip6/::/udp/54321/quic-v1",
      "/ip6/::/udp/54321/quic-v1/webtransport"
    ]
  },
  "AutoConf": {},
  "AutoNAT": {},
  "AutoTLS": {},
  "Bitswap": {},
  "Bootstrap": ["auto"],
  "DNS": {
    "Resolvers": {
      ".": "auto"
    }
  },
  "Datastore": {
    "BlockKeyCacheSize": null,
    "BloomFilterSize": 0,
    "GCPeriod": "1h",
    "HashOnRead": false,
    "Spec": {
      "mounts": [
        {
          "mountpoint": "/blocks",
          "path": "blocks",
          "prefix": "flatfs.datastore",
          "shardFunc": "/repo/flatfs/shard/v1/next-to-last/2",
          "sync": false,
          "type": "flatfs"
        },
        {
          "compression": "none",
          "mountpoint": "/",
          "path": "datastore",
          "prefix": "leveldb.datastore",
          "type": "levelds"
        }
      ],
      "type": "mount"
    },
    "StorageGCWatermark": 90,
    "StorageMax": "10GB"
  },
  "Discovery": {
    "MDNS": {
      "Enabled": false
    }
  },
  "Experimental": {
    "FilestoreEnabled": false,
    "Libp2pStreamMounting": false,
    "OptimisticProvide": false,
    "OptimisticProvideJobsPoolSize": 0,
    "P2pHttpProxy": false,
    "UrlstoreEnabled": false
  },
  "Gateway": {
    "DeserializedResponses": null,
    "DisableHTMLErrors": null,
    "ExposeRoutingAPI": null,
    "HTTPHeaders": {},
    "NoDNSLink": false,
    "NoFetch": false,
    "PublicGateways": null,
    "RootRedirect": ""
  },
  "HTTPRetrieval": {},
  "Identity": {
    "PeerID": "12D3KooWCwKzba6nE1KW6YrDfjWGGAYspZAP6unutUULtpYXmMXC"
  },
  "Import": {
    "BatchMaxNodes": null,
    "BatchMaxSize": null,
    "CidVersion": null,
    "HashFunction": null,
    "UnixFSChunker": null,
    "UnixFSDirectoryMaxLinks": null,
    "UnixFSFileMaxLinks": null,
    "UnixFSHAMTDirectoryMaxFanout": null,
    "UnixFSHAMTDirectorySizeThreshold": null,
    "UnixFSRawLeaves": null
  },
  "Internal": {},
  "Ipns": {
    "DelegatedPublishers": ["auto"],
    "RecordLifetime": "",
    "RepublishPeriod": "",
    "ResolveCacheSize": 128
  },
  "Migration": {},
  "Mounts": {
    "FuseAllowOther": false,
    "IPFS": "/ipfs",
    "IPNS": "/ipns",
    "MFS": "/mfs"
  },
  "Peering": {
    "Peers": null
  },
  "Pinning": {
    "RemoteServices": {}
  },
  "Plugins": {
    "Plugins": null
  },
  "Provider": {},
  "Pubsub": {
    "DisableSigning": false,
    "Router": ""
  },
  "Reprovider": {},
  "Routing": {
    "Methods": {
      "find-peers": {
        "RouterName": "HttpRouterNotSupported"
      },
      "find-providers": {
        "RouterName": "HttpRoutersParallel"
      },
      "get-ipns": {
        "RouterName": "HttpRouterNotSupported"
      },
      "provide": {
        "RouterName": "HttpRoutersParallel"
      },
      "put-ipns": {
        "RouterName": "HttpRouterNotSupported"
      }
    },
    "Routers": {
      "HttpRouter1": {
        "Parameters": {
          "Endpoint": "http://127.0.0.1:19999"
        },
        "Type": "http"
      },
      "HttpRouter2": {
        "Parameters": {
          "Endpoint": "http://127.0.0.1:19998"
        },
        "Type": "http"
      },
      "HttpRouterNotSupported": {
        "Parameters": {
          "Endpoint": "http://kubohttprouternotsupported"
        },
        "Type": "http"
      },
      "HttpRoutersParallel": {
        "Parameters": {
          "Routers": [
            {
              "IgnoreErrors": false,
              "RouterName": "HttpRouter1",
              "Timeout": "5s"
            },
            {
              "IgnoreErrors": false,
              "RouterName": "HttpRouter2",
              "Timeout": "5s"
            }
          ]
        },
        "Type": "parallel"
      }
    },
    "Type": "custom"
  },
  "Swarm": {
    "AddrFilters": null,
    "ConnMgr": {},
    "DisableBandwidthMetrics": false,
    "DisableNatPortMap": false,
    "RelayClient": {},
    "RelayService": {},
    "ResourceMgr": {},
    "Transports": {
      "Multiplexers": {},
      "Network": {},
      "Security": {}
    }
  },
  "Version": {}
}

Description

When using kubo 0.37.0 with a custom routing configuration containing offline HTTP routers, the kubo add operation succeeds instead of failing as expected.

What I was doing:

Adding content to IPFS using kubo add with a configuration that has parallel HTTP routers pointing to offline endpoints (127.0.0.1:19999 and 127.0.0.1:19998). The routers are configured with IgnoreErrors: false and 5-second timeouts.

Expected behavior:

The add operation should fail with an error when all HTTP routers are unreachable, since IgnoreErrors is set to false.

Actual behavior:

The add operation completes successfully and returns a valid IPFS hash despite all routers being offline.

Steps to reproduce:

1. Clone https://github.com/Rinse12/kubo_add_not_throwing_if_providers_not_online
2. Run npm i
3. Run node test-offline-routers.js

The test script automatically configures kubo 0.37.0 with offline HTTP routers and demonstrates the issue. It uses custom routing with parallel routers where both endpoints are intentionally unreachable to simulate network failures.

Configuration details:

• Routing type: custom with parallel HTTP routers
• Router endpoints: offline (127.0.0.1:19999, 127.0.0.1:19998)
• IgnoreErrors: false (failures should not be ignored)
• Timeout: 5s per router

[hsanjuan]

The providing is async, as it is much much slower than the adding itself. So the errors would not be visible. Did it behave as you expect it with the previous version?

[Rinse12]

Did it behave as you expect it with the previous version?

I did not test on other versions. Any versions in mind I should test?

The providing is async, as it is much much slower than the adding itself

In our plebbit-js, we've had the assumption that after ipfs add the content is "online" since it's basic PUT request to the http router.

So the errors would not be visible
Is there any way we can plug into the providing system to see if cid has successfully been announced or not? If not, why did it fail and for which routers etc.

Maybe we could have a new flag in ipfs add to await until the content is announced on the http routers before resolving? It would certainly be useful for our usecase.

[hsanjuan]

Why don't you complement your add with a call to ipfs routing provide -r after it? I think that might do what you need, but test it at least.

Any versions in mind I should test?

v0.36.0 and v0.32.0

[Rinse12]

For routing provide -r I assume it provides immediately regardless of the CID's position in the queue correct? That should work for our usecase.

v0.36.0 and v0.32.0

Same behavior as 0.37

[hsanjuan]

v0.36.0 and v0.32.0

Same behavior as 0.37

ok, so this is not a behaviour change with the releases that have touched the providing code.

I assume it provides immediately regardless of the CID's position in the queue correct?

I'm not sure, but I think yes, I think it is a sync operation as it plugs directly into the Router.Provide rather than the Reprovider (the reprovider has queues and does it async, this should bypass it). Try it please.

[Rinse12]

I'm not sure, but I think yes, I think it is a sync operation as it plugs directly into the Router.Provide rather than the Reprovider (the reprovider has queues and does it async, this should bypass it). Try it please.

We tried it and it works perfectly. We're bundling ipfs add with ipfs routing provide atm to make sure content is provided before publishing an IPNS.