Discrepancy between requested and observed fee rate from fundrawtransaction

[asoltys]

I have some code that passes a custom feeRate parameter to fundrawtransaction. I've noticed that the fee rate on the tx after it's been signed tends to be higher than the provided feeRate though.

Here's an example of a tx after signing that I got from fundrawtransaction after passing in 0.00000100 as the fee rate:

{
  "txid": "d863131bbe3c7789686c2886fcbb15c943c143e332a5c673c7d83e6a0c296048",
  "hash": "02e51dc7c7eb7337ff79d13066adfa00404b35d86712f49e3d8b59cf7872587f",
  "wtxid": "02e51dc7c7eb7337ff79d13066adfa00404b35d86712f49e3d8b59cf7872587f",
  "withash": "2a044674f606a91b309287711d5b075feefa6fe726078c2cd885e0ff1b982cc5",
  "version": 2,
  "size": 1202,
  "vsize": 704,
  "weight": 2816,
  "locktime": 0,
  "vin": [
    {
      "txid": "1a8b4890e803ed3ebf6149143a23089e0f2b1aa2829378aca53e4068905442ec",
      "vout": 0,
      "scriptSig": {
        "asm": "0014b21257d17d3eac63ae75f264ee56c15581978967",
        "hex": "160014b21257d17d3eac63ae75f264ee56c15581978967"
      },
      "is_pegin": false,
      "sequence": 4294967293,
      "txinwitness": [
        "3044022030d0d30b9aaaceef26b53f114d433e1f6de992862ba8c9578106f01e9464806302207a0b03751050201a149748f4f556811f9ec92d45383d307c6a464e2b9c1f79a301",
        "039b162d2ac2d5a93b674286c1abe9bb9e829fd717bfb698f1a483465e83eb86e9"
      ]
    },
    {
      "txid": "5b6fc03954548efd1e7c2dfda16e4dd4fa5a6b9ac4c2c6ff3e32218b879b68f1",
      "vout": 0,
      "scriptSig": {
        "asm": "0014242bf1c66cbdbe3a7236da7c98bb3750d1f6b511",
        "hex": "160014242bf1c66cbdbe3a7236da7c98bb3750d1f6b511"
      },
      "is_pegin": false,
      "sequence": 4294967293,
      "txinwitness": [
        "30440220389b948003d542e7620219399f2ff03a5f07d52628654f83e63fcd988e21563b02207eaf75e922ddca02a138a6e5574529611843ecfe9bb6263ef95283023f0bda3801",
        "0256c083bc0d6547d7fcd60fb493e39b2a9d0b7ddeef22ef06550d2ac695cb7326"
      ]
    },
    {
      "txid": "09dbb3172bf6ee26ff62208434113789adddd50fb5c7e379ac77ab835b565d07",
      "vout": 0,
      "scriptSig": {
        "asm": "0014c3693b9bebcab2a864fd85a41301eef5483a37bb",
        "hex": "160014c3693b9bebcab2a864fd85a41301eef5483a37bb"
      },
      "is_pegin": false,
      "sequence": 4294967293,
      "txinwitness": [
        "304402207b6ba691a6802f8dd9072f60ed17f00bbf64108348ea7fe16e5170b32edc88e0022012e9bd5a60ac81040c02fe39f5b1a22da5acf7946d9ccb15fbd6528ba62b464901",
        "038e4c62e6b3095f62ae316df01b10786589795a7b163063e4c95f9efddda6eafd"
      ]
    },
    {
      "txid": "b0bf648ee962961964ad207ee87c4461787b9612e0719f2f8fa6c3f35336bc45",
      "vout": 0,
      "scriptSig": {
        "asm": "00146e8ca5829872d70f8561687d25d61359f3c952bc",
        "hex": "1600146e8ca5829872d70f8561687d25d61359f3c952bc"
      },
      "is_pegin": false,
      "sequence": 4294967293,
      "txinwitness": [
        "304402203855b3f6955cf142844a7e0abc4f165fe745c1279448f687ac93fd7930aee2c002205468167f64585143d8feb559d16e875795642e6460f82fc48f3bfb88c4f618e501",
        "03d0005cdc368779cfb4bb32e6c4d3ec39f8ecf6cf3088ceadef7cf133eb2376ce"
      ]
    },
    {
      "txid": "0ea1d4a66ad7509344d42dcdad8becc878462d567155893030afd4046995e445",
      "vout": 0,
      "scriptSig": {
        "asm": "00147fb91ddd404d61f105817f24a6e7f3f5601b9f90",
        "hex": "1600147fb91ddd404d61f105817f24a6e7f3f5601b9f90"
      },
      "is_pegin": false,
      "sequence": 4294967293,
      "txinwitness": [
        "304402205a60be940d8470d15440e806352faf690f5a151df3c25d04960cf849a0244ebb02200f1fb4ef567caa0409d0352ca4aa8d76707bdc73821ce9bae3144d894077d79901",
        "03a62c590da8ec8d60cbe25243be8c0ec7db8d35346970062702795135e2691bd8"
      ]
    },
    {
      "txid": "e1e0269825326cb1c6cf5d5cf30564b9dba1f9532d101bb216cb6c4607406dc4",
      "vout": 0,
      "scriptSig": {
        "asm": "0014b518e9ecf2bd2f99da92309554b742899dc96000",
        "hex": "160014b518e9ecf2bd2f99da92309554b742899dc96000"
      },
      "is_pegin": false,
      "sequence": 4294967293,
      "txinwitness": [
        "3044022045c9592aedb8c6fc2c0ccefca7b35756437e1de68a89be29b547505a615c47a402205b42d936a001ae6a35298f41f168a92ae5490423dfe93466ec6b2f9f9161cb2101",
        "0369495abcb8456daf477062ecb9ae42f72a001ced66b9a4fb0698882e24716888"
      ]
    }
  ],
  "vout": [
    {
      "value": 0.00073025,
      "asset": "6f0279e9ed041c3d710a9f57d0c02928416460c4b722ae3457a11eec381c526d",
      "commitmentnonce": "02533c9b577e7b6b3d770bd7ec6e578d2cf3aa9e46ab2f3ff4c311de846bfd27b9",
      "commitmentnonce_fully_valid": true,
      "n": 0,
      "scriptPubKey": {
        "asm": "OP_HASH160 97156b35fd9b105179bd2f0c806fd21b9008dbb2 OP_EQUAL",
        "hex": "a91497156b35fd9b105179bd2f0c806fd21b9008dbb287",
        "reqSigs": 1,
        "type": "scripthash",
        "addresses": [
          "GvxP6QVmBDwo6aKpyNstirSUfT7rubNpRo"
        ]
      }
    },
    {
      "value": 0.00000189,
      "asset": "6f0279e9ed041c3d710a9f57d0c02928416460c4b722ae3457a11eec381c526d",
      "commitmentnonce": "",
      "commitmentnonce_fully_valid": false,
      "n": 1,
      "scriptPubKey": {
        "asm": "",
        "hex": "",
        "type": "fee"
      }
    }
  ]
}

If the observed rate is fee * 1000 / vsize = 0.00000267 btc/kb then that's more than the 0.000001000 btc/kb that was asked for. Is this calculation missing something then? Or is fundrawtransaction perhaps not properly estimating what the signed vsize will end up being?

Relevant stackexchange question where sipa explained to me that the rate is estimated for the final signed tx rather than the unsigned one that fundrawtransaction returns: https://bitcoin.stackexchange.com/questions/95110/calculating-fee-rate-of-tx-returned-by-fundrawtransaction/95117?noredirect=1#comment108765_95117

[stevenroose]

Hmm, that's curious indeed. I've been doing some experiments with fundrawtransaction and feeRate as well and after some errors on the part of my setup, I found that the actual feerate usually was similar to the feerate I asked for, up to some rounding one-off.

But I did also bump into some issues: bitcoin/bitcoin#18475.

[stevenroose]

Could you perhaps test this again with elements built manually from the elements-0.18 branch? We're gonna release a new version with improved fee estimation there soon. It also improves tx size estimation in the fee-setting process a bit.

[asoltys]

Sorry to say but it seems to be worse than before on the elements-0.18 branch 😓

{
  "txid": "bed115ce7e35f9a3e2b522e80c7bae3d24004a0a7c2e8f767e4b42a0cf365e43",
  "hash": "34f7ca3cd8be2aaf95186816642604e8a37c48cb53cf65397dda076f6708d69b",
  "wtxid": "34f7ca3cd8be2aaf95186816642604e8a37c48cb53cf65397dda076f6708d69b",
  "withash": "caf195c29894282505dbcf9af36f526b07a6689b5b8b7ebf0dbf1ea983bef837",
  "version": 2,
  "size": 433,
  "vsize": 346,
  "weight": 1384,
  "locktime": 0,
  "vin": [
    {
      "txid": "da9fb58e4d52dea68d5e1dcd4918f91dad850e814665f1384f438df349fc79fb",
      "vout": 1,
      "scriptSig": {
        "asm": "0014d6aa391a240f3ab8cee6f381406bd0ab341b8316",
        "hex": "160014d6aa391a240f3ab8cee6f381406bd0ab341b8316"
      },
      "is_pegin": false,
      "sequence": 4294967293,
      "txinwitness": [
        "3044022020b33d687b077030797269b2c5ef4c009e980ba8cffa87a2bc4267f2548b0bb9022008fc9ef4fec24e0a4ee04105a46350030604ae77e0a13c57145550dc12959f3c01",
        "02b07574d467215265ee3d9e8a574b17067c8b8d0618c8ee15df839c8f8c7258da"
      ]
    }
  ],
  "vout": [
    {
      "value": 0.00048824,
      "asset": "6f0279e9ed041c3d710a9f57d0c02928416460c4b722ae3457a11eec381c526d",
      "commitmentnonce": "021ab22ca0747e0f558bcff5daad712d9d44799899562b9e2b78488b310afd9582",
      "commitmentnonce_fully_valid": true,
      "n": 0,
      "scriptPubKey": {
        "asm": "OP_HASH160 783b9d6649bdd6420abcba0a9272d1b50fd48dc3 OP_EQUAL",
        "hex": "a914783b9d6649bdd6420abcba0a9272d1b50fd48dc387",
        "reqSigs": 1,
        "type": "scripthash",
        "addresses": [
          "Gt9FuNFaGJarDFRFxQBUx39Ho1gofHRu5e"
        ]
      }
    },
    {
      "value": 0.28831785,
      "asset": "6f0279e9ed041c3d710a9f57d0c02928416460c4b722ae3457a11eec381c526d",
      "commitmentnonce": "03f40b7d37648fc94097c991568397520888db3e5a0cde2c7926ce1417512e2664",
      "commitmentnonce_fully_valid": true,
      "n": 1,
      "scriptPubKey": {
        "asm": "OP_HASH160 a998dbc43f183a93b7b3d6c4325333fec191d5d2 OP_EQUAL",
        "hex": "a914a998dbc43f183a93b7b3d6c4325333fec191d5d287",
        "reqSigs": 1,
        "type": "scripthash",
        "addresses": [
          "GxeGiStvrRkVYB2D8xQ91z5CMi42xtbnmt"
        ]
      }
    },
    {
      "value": 0.00000251,
      "asset": "6f0279e9ed041c3d710a9f57d0c02928416460c4b722ae3457a11eec381c526d",
      "commitmentnonce": "",
      "commitmentnonce_fully_valid": false,
      "n": 2,
      "scriptPubKey": {
        "asm": "",
        "hex": "",
        "type": "fee"
      }
    }
  ]
}

requested: 0.00000100
observed: 1000 * 0.00000251 / 346 = 0.00000725

[stevenroose]

Oh but are you reporting the "observed feerate after fundrawtransaction"? Because fundraw will take into account the output witness size of the rangeproof for outputs that have a valid commitment nonce (i.e. that are ready to be blinded).

In fact that calculation was improved in the new version, so that's why the fee is higher. Before it would undershoot the fee because it was not calculating for the added rangeproof size when blinding.

So it seems like your first 2 outputs are ready to be blinded. Are you not blinding them?

If you're not blinding them, you should not have the nonce field set to the blinding key. How did you create the tx? Using a confidential address?

[asoltys]

Whoops, thanks for catching that. I was blinding the tx when actually sending it out but not during my fee estimation routine. Sure enough when I blind it, the fee rate comes out right. Sorry for troubling you!

[stevenroose]

Yeah so the procedure is createraw -> fundraw -> blindraw -> signraw -> sendraw. It's a bit tedious, I know. But the hard part is that the fundraw needs to estimate the number of bytes that will be added by blindraw and signraw because it needs to add a good fee. Luckily blinding usually adds a fixed amount of bytes per output and the bytes for signing can be well estimated for inputs under management by the wallet.

But if you're not actually blinding the tx, then fundraw will of course over-estimate the size. Currently the intention to blind is indicated by putting the blinding pubkey in the tx "confidential/commitment nonce" field. That field is filled by createraw when you're sending to a blinded address and filled by fundraw on change outputs (IIRC) if the inputs are blinded.

So if createraw fills the nonce fields and you don't want to blind the tx, you can manually remove the nonce fields.

Something like this (not tested):

elements-cli createrawtransaction ... | hal elements tx decode | jq '.outputs.[].nonce = {"type": "null"}' | hal elements tx create