Internet Computer Wiki - User contributions [en]

Node Deployment Guide (with an HSM)

2025-03-31T17:07:35Z

Jamongeon: typo

This runbook covers all steps necessary to install the Internet Computer Operating System (IC-OS) using the legacy NitroKey HSM instructions. To use the non-HSM onboarding instructions, follow the [[IC-OS Installation Runbook]].

The physical machine is expected to be racked and stacked according to its respective manual.

To complete these steps, you are expected to be physically present in the data center your machine(s) reside(s). Once you successfully onboarded your first node, you can bring up the other nodes in parallel.

If you encounter issues during any of these steps, consult the [[Troubleshooting Node Deployment Errors]] page.

⚠️ DFINITY does '''not''' offer live support for Node Providers attempting to deploy nodes.

==1. Choose onboarding path (HSM vs no HSM)==
If you chose the [[Node Provider Onboarding#5. Choose onboarding path .28HSM vs no HSM.29|HSM Node Provider Onboarding Path]], '''continue to the next step.'''

If you chose to onboard '''without''' a Nitrokey HSM, follow the [[IC-OS Installation Runbook]] to onboard your nodes.

==2. Obtain requirements==
*A USB (3.0 speed that can hold at least 4GB) to put the image file on.
** Faster USBs will allow the process to go much faster.
*The NitroKey HSM for your data center.
*[Optional] A USB hub
** This is helpful at some data centers for simultaneously connecting keyboard, mouse, Nitrokey, etc.
*It is recommended that each server has a label with the BMC's MAC address for ease of identification in future dashboard upgrades.

==3. Download installation image==
Download the latest release of the '''IC-OS USB Installer Image''' and the '''corresponding checksum''' from the [https://dashboard.internetcomputer.org/releases Internet Computer Dashboard Releases].
*'''Note that you should always use a release from the last 6 weeks (newer is better) in order to ensure that your node can correctly connect to the network.'''

==4. Verify checksum and unarchive file==
===Mac OS X===
#Open the Terminal and type:
#:<syntaxhighlight lang="shell">shasum -a 256 ~/Downloads/disk-img.tar.zst</syntaxhighlight>
#Compare the calculated checksum with the '''IC-OS installation image checksum''' file downloaded in the previous step. '''Warning:''' Only continue if they are identical, otherwise please post your issue in the [[Node Provider Matrix channel]].
#:Open the Terminal and type: <syntaxhighlight lang="shell">tar xzvf ~/Downloads/disk-img.tar.zst</syntaxhighlight>

===Linux / Ubuntu===
#Open the Terminal and type:
#:<syntaxhighlight lang="shell">sha256sum ~/Downloads/disk-img.tar.zst</syntaxhighlight>
#Compare the calculated checksum with the '''IC-OS installation image checksum''' file downloaded in the previous step. '''Warning:''' Only continue if they are identical, otherwise please post your issue in the [[Node Provider Matrix channel]].
#:Open the Terminal and type: <syntaxhighlight lang="shell">tar xzvf ~/Downloads/disk-img.tar.zst</syntaxhighlight>

===Windows===
#Open PowerShell and type:
#:<syntaxhighlight lang="shell">Get-FileHash -Algorithm SHA256 .\Downloads\disk-img.tar.zst</syntaxhighlight>
#Compare the calculated checksum with the '''IC-OS installation image checksum''' file downloaded in the previous step. '''Warning:''' Only continue if they are identical, otherwise please post your issue in the [[Node Provider Matrix channel]].
#:Open PowerShell and type: <syntaxhighlight lang="shell">tar xzvf .\Downloads\disk-img.tar.zst</syntaxhighlight>

==5. Create Bootable USB Stick==
===Mac OS X===
#Open the Terminal and type:
#:<syntaxhighlight lang="shell">diskutil list</syntaxhighlight>
#All available drives should be shown. Identify which device corresponds to your USB stick. You may need to unmount the USB drive:
#:<syntaxhighlight lang="shell">sudo diskutil unmount /dev/YOUR_USB_DEVICE_MOUNTED_PARTITION # E.g. /dev/disk4s1</syntaxhighlight>
#Replace ''/dev/YOUR_USB_DEVICE'' with the device that corresponds to your USB stick. Additionally, replace the path to your downloaded IC-OS ''disk.img'' file. '''Warning:''' You risk losing your own data if you specify a wrong drive.
#:<syntaxhighlight lang="shell">sudo dd if=/Users/YOUR_USER_NAME/Downloads/disk.img of=/dev/YOUR_USB_DEVICE bs=1M status=progress</syntaxhighlight>If you get a “device is busy” error from the dd command, you can try running the following command to unmount all of the partitions on the disk, then re-run the dd command:
#:<syntaxhighlight lang="shell">sudo diskutil unmountDisk /dev/YOUR_USB_DEVICE # E.g. /dev/disk4</syntaxhighlight>

===Linux / Ubuntu===
#Open the Terminal and type
#:<syntaxhighlight lang="shell">blkid</syntaxhighlight>
#All available drives should be shown. Identify which device corresponds to your USB stick. You may need to unmount the USB drive:
#:<syntaxhighlight lang="shell">sudo umount /dev/YOUR_USB_DEVICE_MOUNTED_PARTITION # E.g. /dev/sdb1</syntaxhighlight>
#Replace ''/dev/YOUR_USB_DEVICE'' with the device that corresponds to your USB stick. Additionally, replace the path to your downloaded IC-OS ''disk.img'' file. '''Warning:''' You risk losing your own data if you specify a wrong drive.
#:<syntaxhighlight lang="shell">sudo dd if=/home/YOUR_USER_NAME/Downloads/disk.img of=/dev/YOUR_USB_DEVICE bs=1M status=progress</syntaxhighlight>

===Windows===
#Download and install [https://rufus.ie/en/ Rufus Portable]
#Start Rufus
#Select the USB stick under device and select the previously downloaded IC-OS disk image and press start
#:[[File:05.png|480px|screenshot]]
#You may see some warnings. Make sure you don't have any other USBs in your computer and chose OK
#:[[File:06.png|480px|screenshot]]
#:[[File:07.png|480px|screenshot]]
#The "Ready" bar will go from left to right as it completes.

==6. Add configuration ==

===A. Open Config.ini in a text editor===

===='''Mac OS X'''====

#Open Finder. You should now be able to see the CONFIG partition. If it's not visible, remove the USB and insert it again.
#:[[File:mac_01.png|580px|screenshot]]
#Double-click <code>config.ini</code> to open it in TextEdit.

===='''Linux'''====

#Open the File Manager. You should now be able to see the CONFIG partition. If it's not visible, remove the USB and insert it again.
#:[[File:linux_01.png|580px|screenshot]]
#Double-click <code>config.ini</code> to open it in KWrite.

===='''Windows'''====

#Open the Disk Management utility with a right click on the Start menu.
#:[[File:09-b.png|300px|screenshot]]#:
#Right click the CONFIG partition.
#Select Change drive letter or paths...
#:[[File:10-b.png|780px|screenshot]]
#Select any letter from the drop-down list.
#:[[File:11-b.png|480px|screenshot]]
# Click OK.
# You should now be able to see the CONFIG partition in your Windows Explorer. Select the <code>config.ini</code> configuration file.
#:[[File:12-b.png|780px|screenshot]]
#Click on Edit to open it.

===B. Edit Config.ini===

Edit the config.ini file to add your network configuration.

Note that '''all Node Providers are requested to deploy two nodes with IPv4 and a domain name for every data center they operate in'''. Node Providers should deploy IPv4 to the '''first two nodes in their first rack'''.
# Set the appropriate value for the node_reward_type.
#:[[File:Pasted Graphic 7.png|780px|screenshot]]
#:You can find the value to set by following [[Node Deployment config.ini]]
#:
#Insert your IPv6 prefix and gateway.
#:[[File:Pasted Graphic 8.png|780px|screenshot]]
#:* The IPv6 prefix should consist of four groups of hexadecimal digits, separated by colons (':'). Each group can contain up to four hex digits.
#:*For example, a valid prefix could look like this: <code>2a00:fb01:400:200</code>
#:*'''Important:'''
#:**The prefix should not have a trailing ':'
#:**IPv6 CIDR notation allows for a double colon ('::') to represent consecutive groups of zeroes in an address. However, the prefix configuration in this context does '''not''' support '::'. The '::' shorthand should '''not''' be used. Even if some groups are all zeros, they must be explicitly written out.
#[Optional] Insert your IPv4 info and domain name.
#:[[File:Pasted Graphic 9.png|780px|screenshot]]
#:*Configuring your node with IPv4 settings is optional, but if you do configure your node with IPv4 settings, you must also define the domain name for your node.
#:*'''Important:'''
#:**Please note that you '''must use a unique IPv4 address for each node you deploy'''. This means that you cannot use a single IC-OS installation image to deploy multiple nodes (like you are able to do when just configuring IPv6 nodes). '''After each IPv4 node deployment, you must plug your IC-OS Installation USB stick back into your laptop and return to [[Node Deployment Guide#6. Add configuration|step 6]] in the node deployment guide to reconfigure your installation image.'''
#:**You can add, remove, or update your node’s IPv4 address and domain name after completing node deployment using dfx commands. See [[Updating your node's IPv4 and domain name|here]] for details.
#Save the changes.
#:*If you have trouble saving this file directly, you may need to save to a known location first, then copy the file into place.
#:*:[[File:mac_03.png|580px|screenshot]]

== 7. Connect Crash Cart==
# In order to configure the UEFI and initiate the installation of the IC-OS, please connect a crash cart to the physical machine.
#Plug-in the VGA/Video, keyboard and IC-OS USB stick
#:[[File:08.png|580px|screenshot]]

==8. UEFI Setup and Boot Menu ==
Use the related page below to set up the BIOS/UEFI according to your hardware vendor.

*[[Node Provider Machine Hardware Guide#Gen 2 Node Machine requirements|Gen2 hardware]]
**[[IC-OS Installation - UEFI Configuration - Gen2 Dell]]
**[[IC-OS Installation - UEFI Configuration - Gen2 Supermicro]]
**[[IC-OS Installation - UEFI Configuration - Gen2 Gigabyte]]
**[[IC-OS Installation - UEFI Configuration - Gen2 ASUS]]
*[[Node Provider Machine Hardware Guide#Gen 1 Node Machine requirements|Gen1 hardware]]
**[[IC-OS Installation - UEFI Configuration - Gen1 Dell|IC-OS Installation - UEFI Configuration - Gen1 Dell (Poweredge R6525)]]
**[[IC-OS Installation - UEFI Configuration - Gen1 Supermicro]]
***
'''Important:''' Do NOT enable the RAID bios setting. Doing so will cause issues with the IC-OS installation.

Resume from this point when you are finished configuring the BIOS.

==9. IC-OS Installation==
#Please wait while the USB Installer is booting up. This process can take up to 3 minutes.
#:[[File:35-sm.png|580px|screenshot]]
#The IC-OS installation starts. Please keep an eye on the progress. This part can take up to 10 minutes. Please remember to check the [[Troubleshooting Node Deployment Errors]] page if you encounter any errors.
#:[[File:36-sm.png|580px|screenshot]]
#Once you get asked to insert the HSM, please remove the keyboard and instead insert the HSM USB device.
#:[[File:37-sm.png|580px|screenshot]]
# If the installation finished successfully, it will initiate a reboot. 🚨 '''Please do not unplug the USB stick or HSM USB''' device at this point. 🚨
#:[[File:38-sm.png|580px|screenshot]]

==10. First Boot==
Please remember to check the [[Troubleshooting Node Deployment Errors]] page if you encounter any errors.

# After the IC-OS installation is complete, the machine reboots and then, after a few minutes, you should see the following log:
#*[[File:Hostos console log.png|600x600px]]
#*This log does '''NOT yet signify a successful onboarding.''' Please wait at least 10 minutes for a <code>Join request successful!</code> log signifying a successful onboarding.
#*If after 10 minutes, you don't see anything else logged to the screen, please '''leave your node running''' and post a message in the Matrix channel with a screenshot of you console and the additional [https://wiki.internetcomputer.org/wiki/Troubleshooting_Node_Deployment_Errors#Support_request_information_requirements Support request information requirements]
#Once you see the <code>Join request successful!</code> message, you may unplug the HSM USB device, USB stick and VGA/Video.
#:[[File:Node join message.png|580px|screenshot]]

Congratulations! Your machine successfully joined the Internet Computer! The machine has joined the IC and the Node Provider will start receiving rewards!

'''Note that if you do NOT see a "Join request successful" message, your node may still have successfully onboarded. Continue to the next step to attempt to verify node onboarding.'''

== 11. Verify node onboarding==

#Obtain your Node ID
#*Your Node ID should have been outputted in the previous step. If it wasn't, '''please wait at least 10 minutes to see if the node ID is logged to the console.'''
#Verify that your node was successfully onboarded by checking its status on the [https://dashboard.internetcomputer.org/ dashboard]
#*The dashboard can be searched by your Node Provider principal. There, you should see the Node ID of your node.
#*If the status of your node is either “Awaiting Subnet” or “Active in Subnet,” '''congratulations! Your machine successfully joined the Internet Computer!'''
#*If the status of your node is NOT either “Awaiting Subnet” or “Active in Subnet”, or if it is NOT listed under your Node Provider principal, you should consult the [[Troubleshooting Node Deployment Errors]] page.
#*:[[File:Node onboarding verification.png|680px|screenshot]]
#If deploying with IPv4, verify that IPv4 was successfully configured
#*Ten minutes after the initial <code>Join request successful!</code> message, you should see another log indicating a successful IPv4 deployment:
#*:[[File:IPv4 log.jpg|680px|screenshot]]
#*If your log says <code>IPv4: none configured</code>, then your IPv4 deployment failed.

If you are failing to verify your node onboarding, consult the [[Troubleshooting Node Deployment Errors]] page.

ICP custody options

2024-12-02T15:49:16Z

Jamongeon:

Understand the benefits and limitations of each custody option so you can choose the wallet that best suits your needs.

{|
|- style="vertical-align:top;"
| style="width: 33%" |
== Onchain Wallets==
Onchain dapps that are easily accessible with the creation of an Internet Identity. Great for daily use and small amount transfers.
* [https://oisy.com/ Oisy]
*[https://nns.ic0.app/ NNS frontend dapp] with [https://identity.ic0.app/ Internet Identity]
*[https://astrox.me/ AstroX ME]
* [https://nfid.one/ NFID Wallet]
*[https://orbitwallet.io/ Orbit] (private alpha)
*[https://www.stoicwallet.com/ Stoic wallet]
| style="width: 34%" |
== Mobile Wallets ==
Mobile apps offer easy access to crypto assets for people who use them frequently.
*[https://plugwallet.ooo/ Plug wallet]
*[https://astrox.me/ AstroX ME]
*[https://klever.io/en-us/crypto-wallet/icp-wallet Klever]
*[https://airgap.it/ AirGap]
*[https://trustwallet.com/ Trust Wallet]
| style="width: 33%" |
==Hardware Wallets==
Maximum security. Hardware wallets hold private keys in airgapped machines or ledger devices.
*[[ICP custody with Ledger Nano|Ledger Nano]] and other Ledger hardware devices
*[https://airgap.it/ AirGap]
*[https://x.com/Tangem/status/1841759127845982370 Tangem]
*[https://www.taurushq.com/blog/taurus-integrates-the-full-icp-value-chain-including-staking-in-its-custody-solution-taurus-protect/ Taurus]
*ICP Custody with [https://github.com/dfinity/quill Quill] (Minimalistic ledger and governance toolkit for cold wallets) Networked computer
*[[ICP custody with seed phrase and air-gapped machine]]
|- style="vertical-align:top;"
|
==Institutional Custody==
For anyone managing large amounts of crypto assets. Institutional custodians offer reliability and customer support.
*[https://www.coinbase.com/custody Coinbase]
*[https://www.sygnum.com/digital-asset-banking/internet-computer-icp/ Sygnum]
* [https://copper.co/en/insights/company-news/copper-adds-support-for-new-token-standards-on-the-internet-computer-blockchain-as-institutional-interest-grows Copper]
*[https://www.crypto-finance.com/ Crypto Finance]
*[https://www.dfns.co/article/icp-support Dfns] (wallet-as-a-service)
*[https://www.primevault.com/ Primevault]
* [https://www.taurushq.com/blog/taurus-integrates-the-full-icp-value-chain-including-staking-in-its-custody-solution-taurus-protect/ Taurus]
*[https://nfid.one/ NFID Wallet]
*[https://orbitwallet.io/ Orbit] (private alpha)
|
==Browser-Extensions==
Great for users already familiar with crypto wallets from other chains.
*[https://plugwallet.ooo/ Plug wallet]
*[https://trustwallet.com/blog/beginners-guide-to-icp Trust Wallet]
*[https://wallet.bitfinity.network Bitfinity Wallet]
*[https://snaps.metamask.io/snap/npm/fort-major/msq/ Metamask] (MSQ snap)
*[https://www.primevault.com/ Primevault] (coupled to institutional custody)
|
==Special Use Cases ==
ICP staking, voting & governance (NNS)

*[https://nns.ic0.app/ NNS frontend dapp]
*[https://www.ledger.com/coin/staking/internet-computer Ledger Nano ICP staking]

SNS staking & governance

*[https://nns.ic0.app/ NNS frontend dapp]
*[https://www.ledger.com/coin/staking/internet-computer Ledger Nano ICP staking]
|}

== Do you want maximum control of your keys?==

===No? Then, use third party custody solutions:===

Third party custody solutions allow you to trade-off ease-of-use for control. The trade-off is simple: if you do not feel comfortable managing your own keys and are willing to have a third party to have access to your keys for the sake of being hand-held through the process, then third party solutions can be the right choice for you.

Third party custody solutions include the Institutional Custody solutions listed above, as well as any of the exchanges supporting ICP on these lists:

*[https://coinmarketcap.com/currencies/internet-computer/#Markets CoinMarketCap list of ICP exchanges]
*[https://www.coingecko.com/en/coins/internet-computer CoinGecko list of ICP exchanges]

===Yes? Then, use self-custody:===
All of the solutions above, except Institutional Custody, are self-custody. On ICP, Oisy and other Onchain Wallets are the preferred solution.

==Self-custody: Maximum ease option==

If you choose self-custody, the simplest option to consider is [https://oisy.com/ Oisy]. And if Staking is required, then [[ICP Custody with NNS frontend dapp]]. It consists of using an [https://identity.ic0.app/ Internet Identity] with the [https://nns.ic0.app/ NNS frontend dapp].

===Traits===
*It's the most convenient, entirely web-based, with no need to download or install anything.
*The [https://nns.ic0.app/ NNS frontend dapp] has all the functionality you need.
*This is a very common method.
*This method is ideal for people who want the easiest path to control their ICP.
*This custody solution has staking and voting built-in if you want to participate in governance.

==Self-custody: Maximum control option==

If you choose self-custody, the options which maximize control are:

*[[ICP custody with seed phrase and air-gapped machine]]
*[[ICP custody with Ledger Nano]]

===Traits===

*This option requires more technical understanding.
*This is the ''safest'' option from a software point of view because the user relies on less software surface area, but it is the ''riskiest'' from a human point of view in that it puts the risk of the user's [[seed phrase]] and [[private key]] custody on them.
*Ideal for people who want as much control over their ICP as possible.
*This custody solution has staking built-in if you want to stake as well.

==See Also==
*[[Tutorials for acquiring, managing, and staking ICP]]

Troubleshooting Failed NNS proposals

2024-07-12T17:08:18Z

Jamongeon: format

=== Background ===
Occasionally, a proposal on the Network Nervous System (NNS) may be adopted but fail during execution. This usually happens due to violations of certain invariants (constraints) in the NNS Registry.

==== Example of a Failed Proposal ====
An instance of such a failure can be seen in https://dashboard.internetcomputer.org/proposal/125578.

==== Screenshot ====
A screenshot of the failed proposal as displayed on the dashboard, for future reference, of https://dashboard.internetcomputer.org/proposal/125578
[[File:Failed-nns-proposal.png|center|thumb|791x791px|Screenshot of a failed NNS proposal in the dashboard]]

=== Identifying the Cause of Failure ===
As of now, the specific reasons for a proposal's failure are not directly accessible on the proposal page. To find out why a proposal failed, you need to interact with the governance canister directly.

==== Steps to Follow ====

# Visit the governance canister page https://dashboard.internetcomputer.org/canister/rrkah-fqaaa-aaaaa-aaaaq-cai.
# Scroll down to the section labeled <code>get_proposal_info</code>
# Use the toggle field to input the proposal number and click "Call".[[File:Failed-proposal-get-info.png|center|thumb|654x654px|Get info on the failed NNS proposal, on the public dashboard]]
# Once the result appears, select "JSON" to view it in Json format.
# The failure reason will be detailed in the output.

==== Example Output ====
Below is an example JSON output for a failed proposal. In this case, the proposal failed because an existing Node Operator record was being duplicated.

Example:

Open that toggle field, enter the proposal number, and click "Call". Once you get the result, you can click on "JSON" to get the Json format, and you can read the failure reason in the output. In this case, the same Node Operator record was already present in the registry when the proposal attempted to re-add the same record.
[
{
"id": [
{
"id": 125578
}
],
"status": 5,
"topic": 5,
"failure_reason": [
{
"error_message": "Error executing ExecuteNnsFunction proposal. Rejection message: IC0503: Canister rwlgt-iiaaa-aaaaa-aaaaa-cai trapped explicitly: Panicked at '[Registry] Verification of the mutation type failed with the following errors: [Registry Canister Error. Msg: Key already present: node_operator_record_mbnsu-w4xfc-pmdok-r2lwo-wxfr4-gigu5-4idcm-5uuuy-znvby-biiny-jqe].', rs/registry/canister/src/registry.rs:293:13",
"error_type": 12
}
],
"ballots": [
[
"14315117116521128082",
{
"vote": 0,
"voting_power": 196565756
}
]
],
"proposal_timestamp_seconds": 1699617467,
"reward_event_round": 0,
"deadline_timestamp_seconds": [
1699963110
],
"failed_timestamp_seconds": 1699878752,
"reject_cost_e8s": 1000000000,
"derived_proposal_information": [],
"latest_tally": [
{
"no": 66258822305525,
"yes": "45221535235544100",
"total": "45541158736625569",
"timestamp_seconds": 1699878752
}
],
"reward_status": 1,
"decided_timestamp_seconds": 1699878752,
"proposal": [
{
"url": "",
"title": [
"Add mbnsu as a Node Operator of Node Provider: 4jjya"
],
"action": [
{
"ExecuteNnsFunction": {
"nns_function": 8,
"payload": "4449444c056c06a795f3ad04018af3a2b108029bf499bd0878bbf187b70c03dddfde9b0d02dba7aaae0d716e716e686d046c020071017901000001011d97289ec1b951d2eceb5cb1e1906a77881899da5298cb6a1c05086e13020a000000000000000001011d78c0a5ff79e51f8d11fa75ad5db01d3d7cee8fecd6658d81c6deb6a20203726731"
}
}
],
"summary": "Node provider “MB Patrankos šūvis” is adding 10 nodes in the rg1 data center"
}
],
"proposer": [
{
"id": "1482125923012887388"
}
],
"executed_timestamp_seconds": 0
}
]

Node Provider Maintenance Guide

2024-07-12T17:05:27Z

Jamongeon: minor grammar fixes

== Troubleshooting ==
See the [[Node Provider Troubleshooting]] guide for info on troubleshooting failed onboardings, unhealthy nodes, networking, and more.

== Submitting NNS proposals ==
As a part of being a Node Provider, you will likely have to submit some NNS proposals. The page at the following link describes some of these proposals: [[Node Provider NNS proposals]]

== Monitoring ==
You are expected to regularly monitor the health of your nodes. Node health status is available on the public dashboard. Example: [https://dashboard.internetcomputer.org/node/235hh-hmjhq-dejel-3q5oi-pdz66-dygbp-yi2sy-zmuiq-rj7r7-65hue-wae node status].

You can also view your node's [https://internetcomputer.org/docs/current/references/node-providers/node-metrics#manually-obtaining-metrics public health metrics] and monitor it with the [https://internetcomputer.org/docs/current/references/node-providers/node-metrics IC observability stack].

===Community Tools and Resources===

Several node providers have generously shared tools to facilitate monitoring node health. These tools can provide notifications in case of node issues.

====Aviate Labs Node Monitor====

*'''Turnkey Solution''': Receive email alerts for unhealthy nodes.
*'''Link''': [https://www.aviatelabs.co/node-monitor AviateLabs Node Monitor]

====DIY Node Monitoring====

*'''GitHub Repository''': Run your own node monitoring system.
*'''Link''': [https://github.com/aviate-labs/node-monitor Aviate Labs GitHub]

====Prometheus Exporter for Node Status====

*'''GitHub Repository''': A tool for exporting node status to a Prometheus-compatible format.
*'''Link''': [https://github.com/virtualhive/ic-node-status-prometheus-exporter IC Node Status Prometheus Exporter]

== Common maintenance tasks ==
*[[Removing a Node From the Registry]]
*[[Adding additional node machines to existing Node Allowance]]
*[[Updating your node's IPv4 and domain name]]
*[[Moving a node from one DC to another]]
*[[iDRAC access and TSR logs]]
*[[Checking node CPU and memory speed]]
*For changing your Node Provider or DC principal, please refer to [[Node Provider NNS proposals]]
*[[Updating Firmware]]
==Permitted tools==
For security and confidentiality reasons, other tools are not allowed to run on the same machine in parallel with the replica. In case you need to troubleshoot an issue, it is recommended to either boot the machine from a USB drive that has a live Linux distribution (e.g. [https://ubuntu.com/tutorials/try-ubuntu-before-you-install#3-boot-from-usb-flash-drive Ubuntu]) or to debug from an auxiliary machine in the same rack on which you have complete control, as described in [[Troubleshooting Unhealthy Nodes#Setting Up an Auxiliary Machine for Network Diagnostics|Unhealthy Nodes#Setting Up an Auxiliary Machine for Network Diagnostics]]

==Scheduled data center outages==
When your data center notifies you of a scheduled outage, you must:

*Notify DFINITY on the [[Node Provider Matrix channel]]
*Make sure your nodes return to one of the healthy statuses when the outage is resolved:
**Active in Subnet - The node is healthy and actively functioning within a subnet.
**Awaiting Subnet - The node is operational and prepared to join a subnet when necessary.
*If a node is degraded at first, give it a little bit of time in case it needs to catch up, but make sure that it does return to one of the two healthy statuses.

==Node rewards based on useful work==
The Internet Computer protocol can tolerate up to 1/3 of nodes misbehaving. There is an ongoing activity to automatically issue node rewards based on useful work, and also to automatically reduce node remuneration in case nodes are misbehaving. This will provide a financial incentive for honest behavior. Please follow the forum and the Matrix channel to stay informed about these activities.

In the meantime, the recommendation is to prepare for this by making sure that your nodes are online and healthy at all times, otherwise you risk penalties even before the automatic node rewards based on useful work become active.

== Subnet recovery==
In case subnet recovery is needed, we may have to reach out to you for assistance. Please make sure you closely follow activities in the Matrix Channel, and enable notifications on new messages -- especially direct mentions.

==General best practices==

# Keep a separate machine in the same rack with appropriate tools for network diagnostics and troubleshooting
# Engage with the node provider community for support and to share effective troubleshooting techniques
===Setting Up an Auxiliary Machine for Network Diagnostics===
Robust Internet connectivity is essential. Without access to internal node logs and metrics, troubleshooting requires alternative strategies, including the use of an auxiliary machine within the same rack. Here's a brief outline for setting up an auxiliary machine in the same rack while following best security practices:

# Hardware Setup:
#* Choose a server with sufficient resources to run diagnostic tools without impacting its performance. There is no need to follow the gen1/gen2 hardware requirements for this server (since this node would not be joining the IC network), but make sure the server is performant enough to run network tests.
#* Ensure that physical security measures are in place to prevent unauthorized access.
# Operating System and Software:
#* Install a secure operating system, like a minimal installation of Linux (we prefer Ubuntu 22.04), which reduces the attack surface.
#* Keep the system updated with the latest security patches and firmware updates.
# Network Configuration:
#* Configure the machine with an IPv6 address in the same range as the IC nodes for accurate testing.
#* Set up a restrictive firewall on the machine to allow ''only the necessary'' inbound and outbound traffic. Consider allowing Internet access for this machine only during troubleshooting sessions and keeping the machine behind a VPN at other times.
# Diagnostic Tools:
#* Install network diagnostic tools such as <code>ping</code>, <code>traceroute</code>, <code>nmap</code>, <code>tcpdump</code>, and <code>iperf</code>.
#* Configure monitoring tools to simulate node activities and track responsiveness.
# Security Measures:
#* Use strong, unique passwords for all accounts, and change them regularly. Or, preferably, do not use passwords at all and use key-based access instead.
#* Implement key-based SSH authentication and disable root login over SSH.
#* Regularly review logs for any unusual activities that might indicate a security breach.
# Maintenance and Updates:
#* Regularly update all software to the latest versions.
#* Periodically test your network diagnostic tools to ensure they are functioning as expected.

==Peer-support and bug reports / resolution: Node Provider Matrix Channel==

Node Providers are encouraged to join the dedicated [[Node Provider Matrix channel]]. This platform can be used for discussing maintenance-related queries and sharing insights, report issues, and search for previous resolutions for operations.

Please consult the Matrix channel for troubleshooting issues '''only after consulting the [[Node Provider Troubleshooting]] guide'''

'''Communication Guidelines on the Matrix Channel'''

As a Node Provider, ensure your notifications are enabled to receive new messages promptly. Your input or intervention might be crucial, especially in urgent situations.

It is recommended to add the node provider name to your alias (handle) on the communication platform, to facilitate communication and enable others to quickly and easily mention you.

Node Provider Troubleshooting

2024-07-12T16:10:51Z

Jamongeon: add missing period

==Specific troubleshooting guides==

* [[Troubleshooting Node Deployment Errors]]
* [[Troubleshooting Unhealthy Nodes]]
* [[Troubleshooting Networking Issues]]
* [[Troubleshooting Failed NNS proposals]]

==Getting the node ID from a node==

# Hook up a console to the node.
# The node ID will print to the screen upon a fresh boot and every 10 minutes thereafter.
# If a node does not show its principal, consult the [[Troubleshooting Node Deployment Errors]] page.

==Node Provider Matrix channel ==
'''After first consulting relevant documentation''', discuss your issue with other Node Providers in the [[Node Provider Matrix channel]].

Node Provider Reward Configuration Guide

2024-07-12T16:08:56Z

Jamongeon: minor grammar/format

After onboarding all your nodes, you must submit a reward configuration proposal in order to start receiving node rewards. If you do not do this, you will not receive node rewards.

In the next code block:

*Replace the <code>NEURON_ID</code> value with your neuron ID from the NNS Frontend Dapp (step 3.6 from the [[Node Provider Onboarding]]).
*Replace the <code>NODE_OPERATOR_PRINCIPAL</code> value with your node operator principal (step 7.1 from the [[Node Provider Onboarding]]).
*Replace <code>NODE_1_MACHINE_ID</code>, <code>NODE_2_MACHINE_ID</code> ... <code>NODE_N_MACHINE_ID</code> with the node machine IDs for all N of your node machines (found on the [https://dashboard.internetcomputer.org/ dashboard])
*Replace <code>NODE_COUNT</code> with the number of nodes you are setting a reward configuration for. *Important : ''If you adding nodes to a previous node allocation or updating a previous node operator record, you have to make sure the total node count should be accounted for'' ''in the'' <code>NODE_COUNT</code> ''parameter.''

==== 1. Create the proposal ====
<syntaxhighlight lang="shell">
$ NEURON_ID = XXXXXXXXXXXXXXXXXXXX
$ NODE_OPERATOR_PRINCIPAL = xxxxx-xxxxx-xxxxx-xxxxx-xxxxx-xxxxx-xxxxx-xxxxx-xxxxx-xxxxx-xxx
$ ./ic-admin \
--nns-url https://ic0.app \
-s ~/.config/dfx/identity/node-provider-hotkey/identity.pem \
propose-to-update-node-operator-config \
--proposer $NEURON_ID \
--summary "Set rewards for the following nodes:

* NODE_1_MACHINE_ID
* NODE_2_MACHINE_ID
* ...
* NODE_N_MACHINE_ID
" \
--node-operator-id $NODE_OPERATOR_PRINCIPAL \
--rewardable-nodes '{"type3.1": NODE_COUNT}'
</syntaxhighlight>

The default reward configuration for Gen2 nodes is <code>type3.1</code>. If a different node configuration is applicable to your node models, replace this value.

Example reward configuration proposal for 3 Gen2 nodes:
<syntaxhighlight lang="shell">
$ NEURON_ID = 13419667327548602649
$ NODE_OPERATOR_PRINCIPAL = uqquy-76uhn-2mys5-dh54e-85ntr-sr953-redif-zkte3-94ndi-st2i1-93t
$ ./ic-admin \
--nns-url https://ic0.app \
-s ~/.config/dfx/identity/node-provider-hotkey/identity.pem \
propose-to-update-node-operator-config \
--proposer $NEURON_ID \
--summary "Set rewards for the following nodes:

* sed94-atzdo-rltqy-tmnhr-fvspg-fat3p-sdbjp-7q3jg-dgfcq-zrlap-cqe
* uq4uy-76uhn-2mys5-dh54e-85ntr-sr953-redif-zkte3-94ndi-st2i1-93t-sdjsl-vjlfn-6duch-vskdu-26pf5-cwibg-zooqk-sdn2e-cgugm-tae
* 39esy-hmrb2-nfvao-t42co-tqfed-y3i7c-xqxyp-idt2w-wgmgr-l4x7l-gae
" \
--node-operator-id $NODE_OPERATOR_PRINCIPAL \
--rewardable-nodes '{"type3.1": 3}'
</syntaxhighlight>

==== 2. Find the proposal on https://dashboard.internetcomputer.org/governance and wait until it has been executed. ====

Validation of Candidate Node Machines

2024-07-12T16:03:13Z

Jamongeon: minor grammar fixes

== Background ==
In order to improve the decentralization network, an optimization model has been proposed on the forum (see [https://forum.dfinity.org/t/ic-topology-series-node-diversification-part-i/23402 node diversification part 1] and [https://forum.dfinity.org/t/ic-topology-node-diversification-part-ii/23553 node diversification part 2]) and also approved (see [https://dashboard.internetcomputer.org/proposal/125367 proposal 125367]). Given a certain target topology, the model optimizes between node rewards (onboarding of additional new nodes and rewards for existing nodes) and decentralization, calculating the minimum number of additional node machines required in order to achieve specific decentralization targets.

The basis for the optimization model is a target IC topology for the next 6 to 12 months which may extend into Q1 2025 for contracts ending for Gen1 node machines. This target topology has been proposed on the forum in [https://forum.dfinity.org/t/ic-topology-node-diversification-part-ii/23553 node diversification part 2], and approved by the community in [https://dashboard.internetcomputer.org/proposal/125549 proposal 125549] on 12th November 2023. This model sets targets for the number of Gen1 nodes and Gen2 nodes per subnet and the decentralization coefficients (Nakamato coefficients) per subnet.
[[File:Validation Candidate Nodes - figure 1.png|center|thumb|800x800px|Validation of Candidate Node Machines - figure 1|alt=]]

Running the optimization tool with the current topology will produce a graph like above, with the red blocks showing the number of additional required node machines in order to reach the decentralization targets set in the IC Target Topology. The number of additional node machines required to reach the decentralization targets is visible in the <code>ObjectiveValue</code> example graph is 68 additional Gen 2 node machines.

With the optimization model and the IC Target Topology, the goal is to implement a transparent, objective and reproducible approach for node provider onboarding. Running the model with the existing IC topology and the number of new node machines intended to be onboarded, the model allows the proposer:

* to verify whether adding additional nodes actually improves the decentralization, and
* allows the community to verify this improvement and vote on the proposal to add additional nodes.

Below the steps are described to run the optimization model and verify any new proposal. An example is also given for a specific (fictional) proposal.

== Basic starting points ==
There are a few basic starting points that everybody should be aware of when using the optimization tool and the IC Target Topology. First of all, using the model requires some basic knowledge of Python and Github, so please make sure to familiarize yourself with these before starting.

In addition, it is important to note that the IC Target Topology is not fixed. It is voted in by the community as the target to be achieved within a certain timeframe. [https://dashboard.internetcomputer.org/proposal/125549 Proposal 125549] describes the target topology for the next half year/year, estimated on the current growth of the IC network. If expected growth should change, the community can decide to vote on an updated IC Target Topology with either more node machines or less node machines. Note that the Target IC Topology is intended to assess effectiveness of adding additional nodes, and identify potential nodes that are not relevant from a decentralization perspective. It does not assess or propose any changes in node rewards, which is a topic that the community discusses and votes upon separately.

If target IC Topology is reached - meaning the <code>Objective Value</code> in the optimization model has reached the value of zero, no new node machines should be onboarded. Hence, NNS proposals for increasing the Node Allowance or defining a new Node Allowance should be rejected by the community. Once a new IC Target topology is defined or (other) Node Providers have reduced their Node Allowance, the <code>Objective Value</code> might increase again which would allow for new proposals to be submitted and approved.
== Steps to follow to assess node relevance ==
Assessing the relevance of adding new node machines using the optimization tool requires you to follow four basic steps:

# Installing and running the latest version of the optimization tool.
# Determining your candidate node configuration.
# Updating your candidate nodes in the optimization tooling.
# If adding new node machines is relevant, submitting the Node Operator proposal with the necessary background description.

Each of these steps is discussed in detail below.

=== Step 1: Downloading and installing the optimization tool ===
The optimization tool can be found in the following Github repository: https://github.com/dfinity/decentralization/. The repository is open sourced to allow the community to help improve the tooling with additional functionality and visualizations.

To run the model. Please follow the following steps:

* Find the repository on https://github.com/dfinity/decentralization/
* Either follow one of the following two approaches:
** Command Line approach - Follow the instructions as described in the README file, i.e.:
*** Clone the repository to you computer
*** Install [https://python-poetry.org/ Python Poetry]
*** Run the model in the command line as described in the README file
** Jupyter Notebook approach - Frequent Python users might have a preference for using [https://jupyter.org/ Jupyter Notebook]. If you want to use Jupyter notebook, you can follow the following steps:
*** Copy the code from the repository files and delete the import commands and main() sections from the separate python files (<code>data_preparation.py</code>, <code>helper_functions.py</code>, <code>linear_solver.py</code>, <code>visualization.py</code>)
*** Run each file in subsequent cells in Jupyter notebook
*** You can subsequently rerun the model and adjust parameters using the interface of Jupyter notebook.
* Carefully read the forum posts on [https://forum.dfinity.org/t/ic-topology-series-node-diversification-part-i/23402 node diversification part 1] and [https://forum.dfinity.org/t/ic-topology-node-diversification-part-ii/23553 node diversification part 2] to understand the working of the tooling.
* Check the outcome of the model without making any updates to the model, in particular the <code>ObjectiveValue</code> as described above. As per 23 November 2023, the Objective Value should be similar to the graph shown above, with an additional 72 nodes needed to reach the target topology.

=== Step 2 : Determine your candidate node configuration ===
The Internet Computer dashboard shows the worldwide distribution of node machines. Based on this distribution, as a potential new Node Provider you can identify potential locations and data center providers for new node machines.

For the Node Provider economics and understanding the costs and rewards for running a node machine, please look at the detailed NP documentation on the Internet Computer wiki pages, in particular those on [[Node Provider Machine Hardware Guide|hardware configuration]], [[Node Provider Data Center and ISP Guide|data center requirements]], and [[Node Provider Remuneration|node rewards]].

Once you have decided on potential data center locations and a potential number of new node machines that have viable economics, you can follow the following steps to add this candidate configuration to the optimization tool:

* Collect the following information:
** The node provider name (if you are an existing node provider, use your existing Node Provider name)
** The data center name (if you are intending to add node machines to an existing data center, use the existing data center name)
** The data center provider name (if you are intending to use an existing data center provider, use the existing data center provider name)
** The country name (if you are intending to add node machines to an existing country, use the existing country name)
** The number of nodes you intend to onboard

=== Step 3: Update the candidate node configuration in the optimization tooling ===
The optimization tool includes a function that allows candidate nodes to be added to the node configuration, and to run the optimization with including these candidate nodes. With the information collected in step 2, the following steps allow for the optimization tool to be run including the candidate nodes:

* Updated the latest topology in the optimization tooling
** Find the <code>df_candidate_nodes</code> function in the <code>main.py</code> of the <code>ic_topology</code> directory.
** You will see the following example entry: <code>df_candidate_nodes = create_candidate_node_dataframe(node_provider ='Lionel Messi',data_center ='Buenos Aires',data_center_provider ='Perron Corporation',country = 'AR',is_sev = True,no_nodes = 0)</code>
** Replace ‘Lionel Messi’ with the Node Provider name.
** Replace ‘Buenos Aires’ with the data center name.
** Replace ‘Perron Corporation’ with the data center provider name.
** Replace ‘AR’ with the country name.
** Replace “0” in no_nodes with the number of nodes intended to be onboarded.
* Run the model and determine the <code>ObjectiveValue</code>:
** If the <code>ObjectiveValue</code> is lower compared to the <code>ObjectiveValue</code> without making changes to the df_candidate_nodes, it means that adding one or more new nodes is increasing the decentralization of the IC. It is important to note that every single node machine should help 1:1 to reduce the <code>ObjectiveValue</code>. For, if the <code>ObjectiveValue</code> without adding any candidate nodes is 68, and 6 candidate nodes are added, the <code>ObjectiveValue</code> should reduce to 68 minus 6 is 62. If the <code>ObjectiveValue</code> is reduced to for example 66, only 2 candidate node machines will support further decentralization of the IC network, and no more than 2 candidate nodes should be added.
** If the ObjectiveValue is the same compared to the <code>ObjectiveValue</code> without making changes to the df_candidate_nodes, it means that adding one or more new nodes does not improve the decentralization of the IC.

=== Step 4: Submitting a Node Operator proposal ===
If the conclusion from step 3 is that adding node machines helps decentralization of the IC network, the final step is to prepare to submit a proposal for onboarding these nodes. Following are the steps to take in order to submit this proposal:

* If you are not yet a Node Provider, please follow the steps on the [[Node Provider Onboarding|NP onboarding wiki page]] to submit a Node Provider Proposal. Please make sure to submit the self-declaration form and the identity-document as part of the proposal, as described in the instructions.
* If you are intending to onboard node machines in a new data center, follow the subsequent steps on the same [[Node Provider Onboarding|NP onboarding wiki page]] to submit a Data Center Proposal.
* Follow the steps on the same [[Node Provider Onboarding|NP onboarding wiki page]] to submit a Node Operator Proposal. The Node Operator Proposal should include the following input in the summary text:
** The number of nodes planned to be onboarded.
** Node provider name (as per the public dashboard).
** The data center and country where nodes are to be onboarded.
** Evidence of running the optimization model and providing reproducible feedback on decrease in <code>ObjectiveValue</code>. This can be done in the following way:
*** Upload a pdf with screenshots of running the optimization model to the wiki, on the page of your specific Node Provider self-declaration documentation.
*** Include the link to this pdf document and the hash of this document in the Node Operator Proposal.
*** Include/describe exactly the used input data and config the optimization model was run with.
*** Clearly state the <code>ObjectiveValue</code> before adding the candidate nodes.
*** Clearly state the <code>ObjectiveValue</code> after adding the candidate nodes.
* Complete the steps of the [[Node Provider Onboarding|NP onboarding wiki page]].

== Example ==
The example below shows how to use the optimization model to validate whether candidate nodes improve the decentralization of the IC network. In the optimization model code in the Github library, you will find a function <code>df_candidate_nodes</code> that is defined as follows:
df_candidate_nodes = create_candidate_node_dataframe(node_provider ='Lionel Messi', data_center ='Buenos Aires', data_center_provider ='Perron Corporation', country = 'AR', is_sev = True, no_nodes = 0)
This function uses a hypothetical new Node Provider, called Lionel Messi, that is interested in setting up new node machines in the Buenos Aires data center. Lionel Messi uses the data center entity Perron Corporation as the data center name. Note that all nodes are marked as <code>is_sev = True</code> which means that Gen2 node machines (with SEV SNP enabled) will be set up, and that <code>no_nodes = 0</code> so when running the current optimization tooling zero nodes will be added; the optimization tool will only use the existing IC topology to calculate the number of required Gen2 Nodes, defined as <code>ObjectiveValue</code>.

Now let’s add one node machine to the candidate node list of Lionel Messi. For this, the function is updated as follows:
df_candidate_nodes = create_candidate_node_dataframe(node_provider ='Lionel Messi', data_center ='Buenos Aires', data_center_provider ='Perron Corporation', country = 'AR', is_sev = True, no_nodes = 1)
Re-running the optimization model will generate (amongst others) the following output. As can be seen from the Node Provider topology Matrix, Lionel Messi is added as a node Provider with one node (which is the top orange block in the graph, and the bottom name in the Node Provider legend on the right).
[[File:Validation of Candidate Node Machines - figure 2.png|alt=Validation of Candidate Node Machines - figure 2|center|thumb|800x800px|Validation of Candidate Node Machines - figure 2]]

The output also shows the data center being added to the Data Center topology. Again, the node is shown on the top (as the block block on second to the top) and the data center name is shown at the bottom of the data center legend as Perron Corporation.

[[File:Candidate Node Machine Validation - figure 3.png|center|thumb|800x800px|Validation of Candidate Node Machines - figure 3|alt=]]
More importantly, the node allocation per subnet is now showing an updated <code>ObjectiveValue</code> with a value of 67, hence 67 more Gen2 node machines are required to reach the IC target topology. Compared to running the model without Lionel Messi’s candidate node machine, the <code>ObjectiveValue</code> is reduced from 68 (see beginning of this article) to 67, hence adding one extra node machine in Buenos Aires support the decentralization of the IC network.

[[File:Validation of Canidate Node Machines - figure 4.png|center|thumb|800x800px|Validation of Candidate Node Machines - figure 4|alt=]]

It would be interesting to validate whether adding additional node machines in Buenos Aires further support the decentralization of the IC. Let’s increase the number of candidate nodes from 1 to 4 by updating the <code>df_candidate_nodes</code> function as follows:
df_candidate_nodes = create_candidate_node_dataframe(node_provider ='Lionel Messi', data_center ='Buenos Aires', data_center_provider ='Perron Corporation', country = 'AR', is_sev = True, no_nodes = 4)
The node allocation per subnet now shows an <code>ObjectiveValue</code> with a value of 64, hence 64 more Gen2 node machines are required to reach the decentralization targets of the IC network. This is a reduction from the original 68 required node machines with 4 nodes. Hence, the model shows that adding 4 candidate nodes in Buenos Aires still adds to the decentralization of the IC network.

Further increasing the number of candidate nodes to 6 shows that decentralization of the IC network still improves. By adding 6 candidate nodes through the updated <code>df_candidate_nodes</code> function, the <code>ObjectiveValue</code> reduces to 62 from 68.
[[File:Validation of Candidate node Machines - figure 5.png|center|thumb|800x800px|Validation of Candidate Node Machines - figure 5]]

Similarly, adding 8 candidate node machines still improves decentralization, as the <code>ObjectiveValue</code> reduced to 60.
df_candidate_nodes = create_candidate_node_dataframe(node_provider ='Lionel Messi', data_center ='Buenos Aires', data_center_provider ='Perron Corporation', country = 'AR', is_sev = True, no_nodes = 8)

[[File:Validation of Candidate node Machines - figure 6.png|center|thumb|800x800px|Validation of Candidate Node Machines - figure 6]]

However, if we increase the number of candidate node machines in Buenos Aires from 8 to 9, we notice that the <code>ObjectiveValue</code> does not reduce anymore. Hence, adding 8 node machines in Buenos Aires is the optimal solution for improving the decentralization targets.
df_candidate_nodes = create_candidate_node_dataframe(node_provider ='Lionel Messi', data_center ='Buenos Aires', data_center_provider ='Perron Corporation', country = 'AR', is_sev = True, no_nodes = 9)
[[File:Validation of Candidate Node Machines - figure 7.png|center|thumb|800x800px|Validation of Candidate Node Machines - figure 7]]

This can be explained by the fact that most subnets (marked in blue in the Node Allocation subnet graph) are already optimized in terms of country limit, node provider limit, and data center limit. Only the subnets with blocks marked in red need additional node machines from unique countries, unique data centers and unique node providers. Adding 8 nodes achieves this. An additional ninth node machine cannot be added to any of the existing subnets in order to improve any of the country limits, data center limits and node provider limits.

== Q&A ==

* What is the maximum number of nodes for one country ?
** As per the agreed target topology ([https://dashboard.internetcomputer.org/proposal/125549 proposal] link) the country subnet limit ranges between 2 and 3. Given that there are currently 40 subnets, once the proposed topology is fully implemented, each country will have the capacity to host between 80 and 120 nodes in total. This range is calculated by multiplying the number of subnets (40) by the per-subnet country limit (2 to 3).

Node Provider Self-declaration

2024-07-12T15:53:30Z

Jamongeon: fix format and add forum post link

This page describes the Node Provider self-declaration process and provides templates. This process and templates are currently in a draft state and discussed by the community (see [https://forum.dfinity.org/t/proposal-for-node-provider-self-declaration/16501 forum post]). Once the NNS has decided on the expected format, the post and this page will be updated to reflect the NNS decision.

== Motivation ==

In a fully decentralized network, the onboarding of a Node Provider is managed entirely by the Network Nervous Systems (NNS). This means that anybody who wants to become a Node Provider needs to submit a proposal that will be voted upon by the community. The main question for the community is: how to decide whether to accept or reject a new Node Provider into the network?

Ultimately, it’s the Node Provider's responsibility to convince the community to adopt their proposal to be added as a new Node Provider. With the self-declaration, the Node Provider shall:

* Establish their identity.
* Make a statement of good intent.

This way, the community will have sufficient information to vote on the Node Provider's onboarding proposal.

== Documents and Templates ==

Before submitting a proposal to become a new Node Provider, the Node Provider shall prepare two kinds of documents. The documents should be delivered as a PDF file that is digitally signed.

=== (1) Self-declaration ===

(A) Statement of identity

Entity name: __________________________________________________

Representative name: __________________________________________

Position within entity: ___________________________________________

Entity address and location: ______________________________________

Country: ______________________________________________________

Chamber of Commerce Nr: _______________________________________

(B) Statement of provision of node machines

I hereby guarantee that I shall provide node machines in accordance with the required Hardware Configuration for running the IC Network, as described on the IC wiki (see [[Node Provider Machine Hardware Guide]]).

(C) Statement of good intent

I guarantee to the world that I shall honestly operate the node machines I provide, and that should I behave dishonestly, for example by deliberately interfering with my node machine(s) to prevent them correctly processing ICP protocol messages, in collusion with others or alone, that I will be liable to users of the network, and to other node providers, for any damages caused.

I further declare I am aware that any deliberate interference with a node machine, which causes it to incorrectly process ICP protocol messages, represents a misuse of that hardware, and of any hardware it interacts with, and that in some jurisdictions, that may constitute a crime.

Signature of representatives

___________________

___________________

___________________

=== (2) Identity Proof ===

The Node Provider shall provide proof that the identity(ies) listed in the self-declaration exist in the real world. The proof can be any document that sufficiently proves the identity of the signers of the self-declaration to the community.

== Process ==

Initially, the process is quite manual. Over time, it shall be automated and for convenience be incorporated into dApps running on the IC. For now:

# '''Preparation:''' The Node Provider prepares:
## The two kinds of documents listed above in a format that is widely available, e.g. PDF. .zip files are not recommended.
### Compute the sha256 hash of each document.
# '''Publication:''' The Node Provider uploads the documents including the sha256 hashes to the wiki page [[Node Provider Self Declarations|Node Provider Self Declarations.]]
# '''Proposal submission:''' The Node Provider submits a proposal to the NNS asking to be accepted to the network.
## The technical instructions are provided on the [[Node Provider Onboarding]] page.
## The summary of the proposal shall point to the published file (step 2) and list the hash (step 1).
# '''NNS vote:''' It’s now up to the NNS community to check whether the provided information matches the community’s expectations and to vote on the proposal.

Node Provider Onboarding

2024-07-12T15:45:59Z

Jamongeon: Update instructions for downloading ic-admin

Learn how to be accepted by the NNS as a Node Provider and onboard your nodes to the IC.

Please allocate up to a week to complete this guide, as it may take several days for a proposal to be accepted by the NNS, and you may have to submit multiple NNS proposals.

'''Note that the following steps do not need to be performed on the node machine itself.''' You can complete them on your personal laptop.

If you encounter issues through any of these steps, check the [[Node Provider Troubleshooting]] page. If that does not solve your problem, you are encouraged to ask for assistance in the [[Node Provider Matrix channel]].

For regular operations after onboarding, please refer to [[Node Provider Maintenance Guide|Node Provider Maintenance Guide.]]

==== ⚠️ DFINITY does '''not''' offer live support for Node Providers attempting to onboard nodes. ====

== '''<big>Requirements</big>''' ==
* Assure your node(s) meet the [[Node Provider Machine Hardware Guide|Node Provider Machine Hardware requirements.]]
* View the [[Node Provider Networking Guide|Node Provider Networking Guide.]]
* Setup a [https://www.ledger.com/ hardware wallet].
*[https://shop.nitrokey.com/shop/product/nkhs2-nitrokey-hsm-2-7/ NitroKey HSM] (Optional, legacy—not recommended).
* 11 ICP (10 of which are to be staked for the NNS proposal deposit).
* Basic understanding of [[Neurons 101|neurons]], [https://internetcomputer.org/docs/current/tokenomics/nns/nns-staking-voting-rewards staking], and [[Governance of the Internet Computer|governance]] proposals, such as understanding what it means to stake a neuron for 8 years.

== 1. Install the required tools ==
===''' A. Install ic-admin '''===

<code>ic-admin</code> is the tool used to create and submit NNS proposals.

==== MacOS ====
# To install <code>ic-admin</code>, view the latest release version in the [https://github.com/dfinity/ic/releases/latest DFINITY/ic repo]. Then, use the following URL <syntaxhighlight lang="shell">
$ curl -L "https://github.com/dfinity/ic/releases/download/[latest-release]/ic-admin-x86_64-darwin.gz" -o - | gunzip > ./ic-admin
$ chmod +x ./ic-admin
</syntaxhighlight>Replace <code>[latest-release]</code> with the most recent IC version, such as <code>release-2024-07-10_23-01-base</code>.
# Verify the binary <syntaxhighlight lang="shell">
$ diff <(shasum -a 256 ./ic-admin | cut -d' ' -f1) <(echo 035abf8925bf54e067d13eee0a6205e883507d6138bcd232ec8069301a9b190a) && echo "ic-admin checksum matches" || echo "***ERROR***: ic-admin checksum does not match"
</syntaxhighlight>

==== Linux ====
NOTE: The instructions below have been tested with the Ubuntu 20.04 release.
# To install <code>ic-admin</code>, view the latest release version in the [https://github.com/dfinity/ic/releases/latest DFINITY/ic repo]. Then, use the following URL <syntaxhighlight lang="shell">
$ curl -L "https://github.com/dfinity/ic/releases/download/[latest-release]/ic-admin-x86_64-linux.gz" -o - | gunzip > ./ic-admin
$ chmod +x ./ic-admin
</syntaxhighlight>Replace <code>[latest-release]</code> with the most recent IC version, such as <code>release-2024-07-10_23-01-base</code>.
# Verify the binary <syntaxhighlight lang="shell">
$ diff <(shasum -a 256 ./ic-admin | cut -d' ' -f1) <(echo e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855) && echo "ic-admin checksum matches" || echo "***ERROR***: ic-admin checksum does not match"
</syntaxhighlight>

===''' B. Install dfx '''===

#<code>dfx</code> is a CLI tool used to generate neuron hotkeys, among other things such as canister deployment and management. <syntaxhighlight lang="shell">
$ sh -ci "$(curl -fsSL https://internetcomputer.org/install.sh)"

</syntaxhighlight>
#Verify that <code>dfx</code> is up to date. <syntaxhighlight lang="shell">
$ export PATH=$HOME/bin:$PATH
$ dfx upgrade
$ dfx --version
</syntaxhighlight>

==2. Create Node Provider hotkey ==

#Create an identity for the Node Provider h'''otkey''' <syntaxhighlight lang="shell">
$ dfx identity new --storage-mode=plaintext node-provider-hotkey

Created identity: "node-provider-hotkey".

$ dfx --identity node-provider-hotkey identity get-principal

xxxxx-xxxxx-xxxxx-xxxxx-xxxxx-xxxxx-xxxxx-xxxxx-xxxxx-xxxxx-xxx
# example node-provider-hotkey: wuyst-x5tpn-g5wri-mp3ps-vjtba-de3xs-w5xgb-crvek-tucbe-o5rqi-mae

</syntaxhighlight>'''You will need the Node Provider hotkey in the next steps.'''

'''Note''': The Node Provider hotkey is NOT the Node Provider principal. This is the hotkey that is used for the NNS proposal submissions only.

'''Note''': You may be prompted to enter a passphrase when creating your identity and accessing your identity principal. Take note of the passphrase you choose.

==3. Create and Manage Neuron via NNS Frontend Dapp and Internet Identity==

#Send at least 11 ICP tokens to your hardware wallet address.
#Navigate to the Neurons tab and create a Neuron by staking at least 10 ICP from your hardware wallet. Staking more ICP is acceptable, but 10 is the minimum needed for this process, and you must have a little more for transaction fees.
#IMPORTANT! Confirm the transaction on your hardware wallet.
#:[[File:-docs-stake_neuron_1.png|1024px|stake neuron]]
#After the Neuron has been created successfully, confirm to "Add NNS Dapp as hotkey" in the dialogue and on your hardware wallet, and close the dialog after the action completes.
#:[[File:-docs-stake_neuron_2.png|1024px|neuron id]]
#Set the dissolve delay to at least 6 months, and confirm the choice in the dialogue and on your hardware wallet. After the action completes, you can close the "Follow Neurons".
#:[[File:dissolve_delay.png|480px|neuron id]]
#You will now see a Neuron listed with its ID. Copy the Neuron ID, since you will need it in the next steps to place the necessary proposals.
#:[[File:Neuron id.png|1024px]]

==4. Add hotkeys==

#Select the Neuron you just created to open Neuron management view and press “Add hotkey” button.
#:[[File:Hotkey 1.png|873x873px]]
# A dialog will pop up where you can enter the hotkey you generated in step 2.1 (output from command <code>dfx --identity node-provider-hotkey identity get-principal</code>). This will allow you to submit NNS proposals using <code>ic-admin</code> and will not be used for anything else. 
#:Press the '''confirm''' button and confirm the transactions on your hardware wallet. 
#:[[File:Hotkey 2.png|899x899px]]
#Get the Ledger Hardware Wallet Principal Id: Navigate back to ICP page and select your Ledger hardware wallet account. You will need to use this Ledger Hardware Wallet principal as the Node Provider principal in order to get the rewards directly into the secure hardware wallet.
#:[[File:Node provider principal 1.png|1024px]]
#:[[File:Node provider principal 2.png|800px]]
#Copy and save this Node Provider principal by clicking on the copy icon after the principal id. You'll need it in the next steps. <syntaxhighlight lang="shell">
$ NODE_PROVIDER_PRINCIPAL=xxxxx-xxxxx-xxxxx-xxxxx-xxxxx-xxxxx-xxxxx-xxxxx-xxxxx-xxxxx-xxx
# Input ledger Hardware Wallet principal, from the NNS FrontEnd dapp https://nns.ic0.app/

# example: $ NODE_PROVIDER_PRINCIPAL=fharn-5vyi2-4xb4a-64yyi-3jpmj-pga23-mxy25-d5uim-fqcro-eoefh-tae
</syntaxhighlight>

==5. Choose onboarding path (HSM vs no HSM)==
Onboarding '''without''' a NitroKey HSM is the recommended onboarding path. In particular, node providers onboarding [[Node Provider Machine Hardware Guide|Gen 2 hardware]] must onboard '''without''' a NitroKey HSM. If you will be onboarding '''without''' a NitroKey HSM, continue to the next step.

If the legacy procedure is necessary, follow the [[NitroKey HSM onboarding instructions]] and '''return to step 8.'''

==6. Setup the Node Operator keys ==
#'''''Ensure dfx is at least version 0.14.''''' Node Operator keys created with older versions of dfx '''will fail to join the IC'''. Run:<syntaxhighlight lang="bash">
$ dfxvm update
$ dfx --version
</syntaxhighlight>
#Create a new principal with dfx:<syntaxhighlight lang="shell">
$ dfx identity new --storage-mode=plaintext node_operator
</syntaxhighlight>
#Confirm <code>node_operator</code> identity was created successfully:<syntaxhighlight lang="shell">
$ dfx identity list
</syntaxhighlight>This list ''should'' contain <code>node_operator</code>.
#Copy new key to a known location:<syntaxhighlight lang="shell">
$ cp ~/.config/dfx/identity/node_operator/identity.pem ./node_operator_private_key.pem
</syntaxhighlight>
#Check the contents of the <code>node_operator_private_key.pem</code> file and double check that it contains the following contents. It is imperative that the first line has <code>-----BEGIN EC PRIVATE KEY-----</code>. If it does not, make sure you use the latest <code>dfx</code> version and that you followed the instructions precisely.<syntaxhighlight lang="shell">
❯ cat ./node_operator_private_key.pem
-----BEGIN EC PRIVATE KEY-----
[3 lines of base64 encoded private key, e.g. n2Nhp68YcQpuS0u96r...]
-----END EC PRIVATE KEY-----
</syntaxhighlight>Note: you must retain access to the <code>node_operator_private_key.pem</code> file for when you onboard nodes in '''[https://wiki.internetcomputer.org/wiki/Node_Provider_Roadmap#Milestone_Five:_Node_Machine_Onboarding roadmap milestone five.]'''

==7. Get the node operator principal==

#Get the principal:<syntaxhighlight lang="shell">
$ NODE_OPERATOR_PRINCIPAL=$(dfx --identity node_operator identity get-principal)
$ echo $NODE_OPERATOR_PRINCIPAL

uqquy-76uhn-2mys5-xxxxx-xxxxx-xxxxx-xxxxx-xxxxx-xxxxx-xxxxx-xxx
</syntaxhighlight>
'''You will need the node operator principal in the next steps.'''

==8. Register your Node Provider principal to the network==
In the next codeblock:
*Replace the <code>NODE_PROVIDER_NAME</code> value with the name of the entity that will provide the nodes.
*Replace the <code>NODE_PROVIDER_PRINCIPAL</code> value with the Ledger Hardware Wallet principal that you got from the NNS Frontend Dapp (step 4.4)
*Replace the <code>NEURON_ID</code> value with your neuron ID from the NNS Frontend Dapp (step 3.6)

*'''''IMPORTANT:''''' Please make sure that you also update the <code>--summary</code> and include a link to the forum discussion, your company's web page, and/or to another place that can convince the voting community that you are making a legitimate request. You must also include the file hash for the [[Node Provider Self-declaration|self declaration and proof of identity documents]], or the proposal will be rejected. This way you will avoid the community voting NO to your proposal and you losing your staked ICPs.

##Create the Proposal <syntaxhighlight lang="shell">
$ NODE_PROVIDER_NAME="My Company"
$ NODE_PROVIDER_PRINCIPAL=xxxxx-xxxxx-xxxxx-xxxxx-xxxxx-xxxxx-xxxxx-xxxxx-xxxxx-xxxxx-xxx
$ NEURON_ID=XXXXXXXXXXXXXXXXXXXX
$ ./ic-admin \
--nns-url https://ic0.app \
-s ~/.config/dfx/identity/node-provider-hotkey/identity.pem \
propose-to-add-or-remove-node-provider add \
--proposer $NEURON_ID \
--proposal-title "Register a node provider '${NODE_PROVIDER_NAME}'" \
--summary "Register a node provider '${NODE_PROVIDER_NAME}', in line with the announcement and discussion at <https://forum.dfinity.org/t/...>. The self-declaration documentation is available at <https://wiki.internetcomputer.org/wiki/...> with SHA256 hash <SHA256>. The proof of identity is available at <https://wiki.internetcomputer.org/wiki/...> with SHA256 hash <SHA256>." \
--node-provider-pid "$NODE_PROVIDER_PRINCIPAL"
</syntaxhighlight>Note: make sure <code>${NODE_PROVIDER_NAME}</code> is presented in single quotes, so the IC dashboard can pick up and display the correct Node Provider name.
#Find the proposal on https://dashboard.internetcomputer.org/governance and '''wait until it is executed before proceeding to next step.'''
#In order to expedite the speed of your proposal's approval, it is best to create a post in this [https://forum.dfinity.org/t/new-node-provider-proposals/16643/69 forum thread] to raise awareness of your proposal. You can use this as a [https://docs.google.com/document/d/1nKy5hKiF72a4NCHvpgij-Np9pbtR5KOBbF1W6qr-nds/edit?usp=sharing template] for the post.

''See guide for [[Troubleshooting Failed NNS proposals]]''

==9. Register your datacenter to the network (if necessary)==
#Search for your data center on https://dashboard.internetcomputer.org/centers.
#*If you found the datacenter that is hosting your nodes, remember its ID, and skip the following section. Otherwise, proceed with the registration of a new DC record.
#:[[File:dc_id.png|1041x1041px|alt=]]
===Create a data center record for a new DC===
In the next block of code:
*Replace the <code>NEURON_ID</code> value with your neuron ID from the NNS Frontend Dapp (step 3.6)
*Replace the JSON fields from the <code>–data-centers-to-add</code> argument and their corresponding values in <code>--summary</code>:
**<code>"id"</code> represents the city that your datacenter is in and is formulated as a combination of two letters representing the city and an incrementing number. Search data center IDs on https://dashboard.internetcomputer.org, and find a combination of two letters and a number that’s not yet registered. Examples:
***dl1 (Dallas, no IDs with “dl” prefix)
***zh10 (Zurich, numbers 0-9 are already registered)
**:[[File:dc_id.png|1024px]]
**<code>"region"</code> represents the local region of a datacenter and is formulated as a three-part string divided by commas. The three parts making the string are continent, country code, and region, in the given order. Examples:
***North America,US,Florida
***Europe,DE,Bavaria
***Asia,SG,Singapore
**:[[File:datacenter_region.png|1024px]]
**<code>"owner"</code> The entity that provides your datacenter facilities.
***Search https://dashboard.internetcomputer.org for existing data center providers.
*** If there’s match, make sure you use the same exact some name for your datacenter.
***Otherwise, name the data center owner to your best knowledge.
**:[[File:datacenter_owner.png|1024px]]
**<code>"gps"</code> GPS coordinates.
***Find your datacenter on https://www.google.com/maps/.
***Right click on location, and select the GPS coordinates (first item in the menu) in order to copy them.
**:[[File:maps.png|310x310px|alt=Getting GPS coordinates|Getting GPS coordinates]]

#Create the proposal: <syntaxhighlight lang="shell">
$ NEURON_ID=XXXXXXXXXXXXXXXXXXXX
$ ./ic-admin \
--nns-url https://ic0.app \
-s ~/.config/dfx/identity/node-provider-hotkey/identity.pem \
propose-to-add-or-remove-data-centers \
--summary "Register a Flexential datacenter as dl1 in North America,US,Texas" \
--skip-confirmation \
--proposer $NEURON_ID \
--data-centers-to-add '{
"id": "dl1",
"region": "North America,US,Texas",
"owner": "Flexential",
"gps": [
33.00803, -96.66614
]
}'
</syntaxhighlight>'''Remember to replace all the values of both the arguments <code>–data-centers-to-add</code> and <code>--summary</code>'''
#Find the proposal on https://dashboard.internetcomputer.org/governance and wait until it's executed before proceeding to next step.
#In order to expedite the speed of your proposal's approval, it is best to create a post in this [https://forum.dfinity.org/t/new-node-provider-proposals/16643/69 forum thread] to raise awareness of your proposal. You can use this as a [https://docs.google.com/document/d/1Hg0tI9O5__Tp4qKrNKuTADsQT7Z47I6aAFXbiDimG_U/edit?usp=sharing template] for the post.

''See guide for [[Troubleshooting Failed NNS proposals]]''

==10. Create a node operator record==
'''''IMPORTANT''':'' Before submitting the Node Operator record, please go through the description of [[Validation of Candidate Node Machines]] to validate whether additional node machines are needed for decentralization of the IC-network.

* Create a pdf with the outcome of running the optimization model and the steps that allow the community to reproduce the validation check.
* Include the pdf in the wiki page that includes the NP documents (self-declaration and proof-of-identity document)
* Include a link to the pdf as well as the hash of the document in the Node Operator record proosal.

In the next codeblock:
*Replace the <code>NEURON_ID</code> value with your neuron ID from the NNS Frontend Dapp (step 3.6).
*Replace the <code>NODE_PROVIDER_PRINCIPAL</code> value with the Ledger Hardware Wallet principal that you got from the NNS Frontend Dapp (step 4.4).
*Replace the <code>NODE_OPERATOR_PRINCIPAL</code> value with your node operator principal (step 7.1). '''Important''': if you are adding an additional node operator record because you will deploy nodes in another data center, please make sure to create a new node operator principal first (steps 6 and 7). A node operator principal can only be tied to 1 data center.
*Replace the <code>NODE_PROVIDER_NAME</code> value with the name of the entity that will provide the nodes.
*Replace the <code>NODE_ALLOWANCE</code> variable value with number of nodes you are providing.
* Replace the <code>DC_ID</code> variable value with id of your datacenter.

# Create the proposal: <syntaxhighlight lang="shell">
$ NEURON_ID=XXXXXXXXXXXXXXXXXXXX
$ NODE_PROVIDER_PRINCIPAL=xxxxx-xxxxx-xxxxx-xxxxx-xxxxx-xxxxx-xxxxx-xxxxx-xxxxx-xxxxx-xxx
$ NODE_OPERATOR_PRINCIPAL=xxxxx-xxxxx-xxxxx-xxxxx-xxxxx-xxxxx-xxxxx-xxxxx-xxxxx-xxxxx-xxx
$ NODE_PROVIDER_NAME="My Company"
$ NODE_ALLOWANCE=8
$ DC_ID=dl1

$ ./ic-admin \
--nns-url https://ic0.app \
-s ~/.config/dfx/identity/node-provider-hotkey/identity.pem \
propose-to-add-node-operator \
$NODE_PROVIDER_PRINCIPAL \
--summary "Node provider '$NODE_PROVIDER_NAME' is adding $NODE_ALLOWANCE nodes in the $DC_ID data center. The result of the canidate node machine validation and exact configuration run is available at <https://wiki.internetcomputer.org/wiki/...> with SHA256 hash <SHA256>." \
--proposer $NEURON_ID \
--node-operator-principal-id $NODE_OPERATOR_PRINCIPAL \
--node-allowance $NODE_ALLOWANCE \
--dc-id $DC_ID
</syntaxhighlight>
#Find the proposal on https://dashboard.internetcomputer.org/governance and wait until it's executed before proceeding to next step.

''See guide for [[Troubleshooting Failed NNS proposals]]''

Node Provider Documentation

2024-07-12T13:51:21Z

Jamongeon: Revise node provider homepage

==Introduction==
The Internet Computer is a decentralized network of nodes running an instance of the network's protocol software. These nodes are owned by Node Providers who invest in and operate the node hardware that powers the network. Running these nodes in data centers provides the high performance and cost-effectiveness of the Internet Computer.

Each Node Provider receives rewards for their nodes' [[Proof of Useful Work|useful work]]. Individuals or organizations can become Node Providers by submitting a proposal to the Network Nervous System (NNS), the Decentralized Autonomous Organization (DAO) that governs the Internet Computer. The ICP community then votes on whether or not to include the Node Provider.

The more diverse the set of Node Providers who supply node machines, the more resilient the Internet Computer is. You can support the Internet Computer and the ICP community by becoming a Node Provider and increasing its decentralization.

== ⚠️ No new node machines being onboarded: Target topology reached ==
Last year, the community voted on an ICP target topology with a maximum number of node machines, as well as on an optimization model to validate candidate node machines against this ICP target topology. The ICP target topology has been reached as of December 2023; see more information in this [https://forum.dfinity.org/t/new-node-provider-proposals/16643/322?u=svenf forum post].

This means that the '''ICP network currently does not require any additional node machines''' in order to reach its decentralization targets. Therefore, DFINITY will not vote to adopt any proposals for new node machines being added to the IC network. Once the subnets are reaching capacity and more subnets are required to run all applications, the community may decide to increase the target topology again and allow additional node machines to join the network.

== Roadmap ==
To become a Node Provider, follow the [[Node Provider Roadmap]], which explain the costs, rewards, responsibilities, and steps required to become a Node Provider.

== Resources ==
*Node deployment guides:
**[[Node Deployment Guide (with an HSM)|Legacy (Gen-1) Node Deployment Guide (with an HSM)]]
**[[Node Deployment Guide|Current (Gen-2) Node Deployment Guide (without an HSM)]]
*Node Provider onboarding
**[[Node Provider Onboarding]]
**[[Node Provider Self-declaration]]
**[[Validation of Candidate Node Machines]]
**[[Node Provider Reward Configuration Guide]]
**[[Troubleshooting Failed NNS proposals]]
*Troubleshooting and maintenance:
**[[Node Provider Maintenance Guide]]
**[[Node Provider Troubleshooting|Node Provider Troubleshooting Guide]]
**[[Node Provider Matrix channel]]
*Guides and resources
**[[Node Provider Machine Hardware Guide]]
**[[Node Provider Networking Guide]]
**[[Node Provider Alerting Options]]
**[[Node Provider Data Center and ISP Guide]]
**[[Node Provider Decentralization and Security Guide]]
**[[Node Provider Legal Guide]]
**[[Node Provider Remuneration]]
**[[Node Provider Domain Name Guide]]
**[[BMC Password Reset Guide]]
**[[Gen1 Node Provider onboarding Gen2 node machines]]
**[[RMU build Gen-1.5|Guide for Gen-1 to Gen-1.5 RMU build]]
*Other resources:
**[[Node Provider FAQ]]

Chain-key Bitcoin

2023-07-19T14:52:10Z

Jamongeon: Removing `track_balance` API endpoint and all mentions of it; fixing bitcoin capitalization

== Overview ==

Chain-key Bitcoin (ckBTC) is a token on the Internet Computer that is backed 1:1 by bitcoin (BTC) such that 1 ckBTC can always be redeemed for 1 BTC and vice versa.

Unlike other tokens pegged to bitcoin, the ckBTC token does not rely on a third-party bridge for the conversion between BTC and ckBTC, making it a substantially more secure alternative to “wrapped” tokens.

While chain-key bitcoin and regular bitcoin have the same value, the advantage of chain-key bitcoin is fast and cheap transfers: A transfer is finalized within a few seconds (a speed-up of roughly three orders of magnitude compared to transfers on the Bitcoin blockchain) and only costs 0.0000001 ckBTC (approximately two orders of magnitude lower than the Bitcoin miner fees).

== Architecture ==

The ckBTC functionality is built upon the [[Bitcoin integration]] of the Internet Computer, which makes it possible for canisters to receive, hold, and send bitcoin.

There are two canisters, the ckBTC minter and ckBTC ledger, that together provide the ckBTC functionality. The ckBTC minter mints new ckBTC tokens whenever it receives bitcoin. Likewise, it burns ckBTC tokens whenever an owner of ckBTC tokens requests a withdrawal of bitcoin. The ckBTC minter needs to receive BTC, based on a large number of confirmations due to the lack of finality in Bitcoin, before it mints the same amount in ckBTC and it burns ckBTC before it transfers BTC back to the users.

A best-effort approach is used for the transfer of BTC to the users, i.e., the user cannot get the burned ckBTC back. Rather, the ckBTC minter repeatedly attempts to transfer BTC out. Note that “user” refers to the caller of the functions exposed by the ckBTC minter and ckBTC ledger in the following. A human user or another canister may be behind the function invocations.

The ckBTC ledger is [https://github.com/dfinity/ICRC-1/blob/main/standards/ICRC-1/README.md ICRC-1 compliant], updating the balance accounts when ckBTC tokens are transferred and executing the mint and burn operations coming from the ckBTC minter.

An overview of the basic architecture is depicted in the following figure.
[[File:CkBTC Specification - Overview (Version 2).png|alt=|center|thumb|600x600px|High-level overview of chain-key Bitcoin.]]
The figure shows the main flow at a high level of abstraction: Users interact with the ckBTC minter and the ckBTC ledger to convert ckBTC/BTC and transfer ckBTC, respectively. The ckBTC minter interacts with the Bitcoin canister to retrieve information about the Bitcoin network state and send Bitcoin transactions.

The ckBTC minter further interacts with the [[Know-Your-Transaction (KYT) Canister|KYT canister]] for "know-your-transaction" (KYT) checks. These checks are meant to ensure that the ckBTC minter only uses "clean" bitcoins to back the issued ckBTC tokens and to prevent transferring bitcoins to Bitcoin addresses that are considered to be associated with illicit activity. As such, these KYT checks provide an additional layer of security to ckBTC users.

== Technical Details ==

=== Bitcoin Addresses ===
Within this page, Bitcoin addresses are implicitly understood to be of type P2WPKH (“pay to witness public key hash”) as defined in [https://en.bitcoin.it/wiki/BIP_0141 BIP-141]. All addresses are rendered in the Bech32 format as defined in [https://github.com/bitcoin/bips/blob/master/bip-0173.mediawiki BIP-173]. The main advantage of P2WPKH over the legacy P2PKH address type is that its use results in lower transaction fees.

For this reason, it is the default address type in most wallets for standard transactions.

=== ckBTC Ledger ===
The ckBTC ledger is a [https://dashboard.internetcomputer.org/canister/mxzaz-hqaaa-aaaar-qaada-cai canister], controlled by the NNS (specifically, the [https://dashboard.internetcomputer.org/canister/r7inp-6aaaa-aaaaa-aaabq-cai NNS root canister]), running on the [https://dashboard.internetcomputer.org/subnet/pzp6e-ekpqk-3c5x7-2h6so-njoeq-mt45d-h3h6c-q3mxf-vpeq5-fk5o7-yae pzp6e] subnet. Since it is ICRC-1 compliant, it offers [https://github.com/dfinity/ICRC-1/blob/aa82e52aaa74cc7c5f6a141e30b708bf42ede1e3/standards/ICRC-1/README.md#methods all functions] defined in the ICRC-1 standard. In particular, it provides the functionality to transfer ckBTC between accounts.

The ckBTC ledger is responsible for keeping account balances and for transferring ckBTC between accounts. It provides the following functionality:

* It enables the ckBTC minter to mint and burn ckBTC.
* It enables the transfer of ckBTC among users.

As mentioned above, the transfer fee is 0.0000001 ckBTC, the equivalent of 10 Satoshi. Since the fee in ckBTC is burned, every transaction leads to a growing surplus in BTC controlled by the ckBTC minter. The surplus will eventually be used to cover the cycle cost of the ckBTC minter.

The minting account is the ckBTC minter’s default account; that is, the account derived from the ckBTC minter’s principal ID and the all-zero subaccount.

The initial supply of the ckBTC ledger is 0. ckBTC tokens are minted only when the ckBTC minter receives bitcoin, ensuring that the ckBTC supply managed by the ckBTC ledger is upper bounded by the amount of bitcoin held by the ckBTC minter.

=== ckBTC Minter ===

The ckBTC minter is a [https://dashboard.internetcomputer.org/canister/mqygn-kiaaa-aaaar-qaadq-cai canister] that is controlled by the NNS and running on the [https://dashboard.internetcomputer.org/subnet/pzp6e-ekpqk-3c5x7-2h6so-njoeq-mt45d-h3h6c-q3mxf-vpeq5-fk5o7-yae pzp6e] subnet as well.

The ckBTC minter is the canister responsible for managing deposited BTC and minting/burning ckBTC based on the amount of deposited BTC. It provides the following functionality:

* For a certain principal ID and an optional subaccount, it returns a specific Bitcoin address under the ckBTC minter’s control.
* Users can inform the ckBTC minter about bitcoins that were sent to an address controlled by the ckBTC minter. If the balance has increased, the ckBTC minter mints the same amount in ckBTC for the user associated with the Bitcoin address.
* Users can ask the ckBTC minter to track the balance of a Bitcoin address until new funds are received, which then automatically triggers the minting of the equivalent amount of ckBTC.
* Users can request to get bitcoins back. The ckBTC minter burns the same amount of ckBTC and transfers the bitcoins to the address provided by the user.

The ckBTC minter canister has a few important configuration parameters including:

* <code>retrieve_btc_min_amount</code>: This is the minimum ckBTC amount that can be burned and, correspondingly, the minimum BTC amount that can be withdrawn. The parameter is set to 0.001 BTC, or 100,000 satoshi.
* <code>max_time_in_queue_nanos</code>: Any BTC retrieval request should be kept in a queue for at most this time. Caching requests rather than handling them right away has the advantage that multiple requests can be served in a single transaction, saving Bitcoin miner fees. The parameter is set to 10 minutes, which corresponds to the expected time between Bitcoin blocks.
* <code>min_confirmations</code>: The number of confirmations required for the ckBTC minter to accept a Bitcoin transaction. In particular, the ckBTC minter does not mint ckBTC before a transaction transferring BTC to a Bitcoin address managed by the ckBTC minter reaches this number of transactions. The parameter was initially set to 72 but has been reduced to 12 in the meantime.
* <code>kyt_fee</code>: The fee that must be paid for KYT checks. It is currently set to 2000 satoshi.
The other parameters are self-explanatory and can be found in the [https://github.com/dfinity/ic/blob/master/rs/bitcoin/ckbtc/minter/ckbtc_minter.did ckBTC minter Candid file].

=== State Management ===

==== Managed Addresses ====
A Bitcoin address that is controlled by the ckBTC minter and has a positive balance is called a managed address. If the balance of a managed address reduces to zero, the address is no longer managed and can be removed from the state.

Note that it is possible to increase the balance again, in which case the address becomes managed again.

==== Unspent Transaction Outputs ====
Once a new Unspent Transaction Output (UTXO) under the control of the ckBTC minter is discovered (using the <code>update_balance</code> function), it is stored internally in a set called <code>available_utxos</code> (defined [https://github.com/dfinity/ic/blob/2348b094d3d27616ee3f049d3048baa1da8d625a/rs/bitcoin/ckbtc/minter/src/state.rs#L305C14-L305C14 here] in the source code).

All discovered UTXOs remain in this set until a Bitcoin transaction is created to spend one or more of them when retrieving bitcoins. When a transaction is created spending some UTXOs, these UTXOs are removed from the set <code>available_utxos</code> and inserted in the <code>used_utxos</code> field of the <code>SubmittedBtcTransaction</code> struct (defined [https://github.com/dfinity/ic/blob/70d19f16c17f8f42987a46d473ba27705927cdb7/rs/bitcoin/ckbtc/minter/src/state.rs#L87 here] in the source code), which is the internal representation of a Bitcoin transaction.

A UTXO is removed from the ckBTC state when the <code>SubmittedBtcTransaction</code> struct that contains the UTXO is removed from the state.

==== Transactions ====
To provide information about current transactions, outgoing transactions must be cached. A transaction is cached until its inputs are no longer considered unspent based on a large number of confirmations.

Every transaction that the ckBTC minter creates has an output that sends the ckBTC minter fee plus the transaction change back to its main BTC address (P2WPKH address derived from its public key with an empty derivation path) using the Bitcoin integration’s <code>get_utxos</code> function.

A transaction can be removed from the cache if the transaction output that belongs to the ckBTC minter appears in the returned list when calling <code>get_utxos</code> with sufficient confirmations for the ckBTC minter’s main BTC address.

The ckBTC minter may resubmit transactions, making use of Bitcoin’s request by fee (RBF) mechanism as defined in [https://github.com/bitcoin/bips/blob/master/bip-0125.mediawiki BIP-125]. In the case of ckBTC, a resubmission adds a transaction to the cache that spends exactly the same UTXOs as the transaction it replaces. The only difference is that the BTC amount sent to the user is reduced in order to increase the fee.

BIP-125 states that at most 100 transactions may be evicted from the mempool, i.e., the fee cannot be increased more than 100 times. Moreover, the fee must be increased at least by the minimum relay fee (see minrelaytxfee [https://en.bitcoin.it/wiki/Miner_fees#Relaying here]) of 1 Satoshi/vbyte.

For example, if we assume a minimum increase of 200 Satoshi (the minimum fee for a basic <code>segwit</code> transaction with one input and one output is 192 Satoshi and the number per output is always lower than 200 if there are at least as many outputs as inputs), the minimum transfer amount should be at least 20,000 Satoshi which equals 0.0002 BTC. When adding a base fee at a large fee rate of 100 Satoshi/vbyte and assuming a virtual transaction size of 200 vbyte per output, we get a minimum transfer amount of 0.0004 BTC. Adding a security margin, we define the minimum retrieval amount to be <code>MIN_RETRIEVAL_AMOUNT = 0.001 BTC</code>.

''To ensure that every transaction can potentially be updated, the RBF flag must be set on every transaction.''

====Know Your Transaction & Fees====
Before UTXOs are accepted, they undergo a know-your-transaction (KYT) check where information about the UTXO is sent to a “KYT canister”, which interacts with KYT service providers, such as Chainalysis, to check if the UTXO is “clean” using HTTPS outcalls.

In a similar fashion, there is a KYT check of the destination address when attempting to transfer bitcoins out of the ckBTC minter.

Both of these KYT checks incur fees, charged in ckBTC:

*The KYT fee is subtracted from the amount to be minted and the amount transferred out.
*The KYT fee is 2000 satoshi per KYT check.

The KYT canister supports multiple KYT access key providers, called maintainers, each having a principal ID and (secret) access key to the external KYT service. Since the maintainers pay for their subscriptions, they must be remunerated. To this end, the ckBTC minter maintains a mapping of maintainers to the owed amount, which is the number of KYT checks that were performed with the respective maintainer’s credentials times the KYT fee of 2000 Satoshi.

Each successful response from the KYT canister contains the principal ID of the maintainer so that the ckBTC minter can update the owed amount correctly. Once every 24 hours, the ckBTC minter pays out the owed amounts in ckBTC to all maintainers.

==Converting BTC to ckBTC==

In this section, the process to convert BTC to ckBTC is explained, making use of the ckBTC minter and ckBTC ledger endpoints.

The first step is for the user to determine the Bitcoin address where the user is supposed to transfer bitcoin for the minting process by calling the <code>get_btc_address</code> endpoint. Next, the user transfers the desired BTC amount to this Bitcoin address.

Internally, the tracking works as follows. Since the expected time between Bitcoin blocks is 10 minutes, the ckBTC minter first checks if new unspent transaction outputs (UTXOs) are available for that address with at least the required number of confirmations after <code>10*min_confirmations</code> minutes. If there is at least one new UTXO, tracking stops and the ckBTC minter instructs the ckBTC ledger to mint the same amount of ckBTC tokens into the account derived from the principal ID and the subaccount.

If no new UTXOs are discovered with sufficiently many confirmations, the ckBTC ledger checks if there are new UTXOs with at least one confirmation. If this is not the case, tracking stops as well. Otherwise, the expected time until the first new UTXO reaches the desired number of confirmations is computed, which is 10 minutes times the difference between the desired number of confirmations and the current number of confirmations. The same process then repeats until ckBTC tokens are minted or tracking stops.

It is evident from this description that it’s possible that tracking may stop before ckBTC tokens are minted, for example, if it takes an unusually long time until the transaction appears in a block. In this case, the endpoint can be invoked again. Alternatively, the user can wait until the transaction has the required number of confirmations and call the <code>update_balance</code> endpoint.

===Technical Details===
As mentioned above, a call to the <code>update_balance</code> endpoint triggers a call to the Bitcoin canister to get UTXOs for the Bitcoin address associated with the given <code>principal-subaccount</code> pair. Let '''''R''''' denote the set of returned UTXOs. The following pseudo-code illustrates how the UTXOs are processed:<syntaxhighlight lang="rust">
for utxo in new_utxos(R): // R = set of received UTXOs in the get_utxos call
if utxo.value >= KYT_FEE:
if utxo in checked_utxos: 
(uuid, state) = checked_utxos.get(utxo)
else:
(uuid, state, kyt_provider) = kyt_canister.check(utxo).await?
if state == clean:
D = D \ {utxo}
P = P ∪ {utxo}
checked_utxos.set(utxo, (uuid, clean))
owed_kyt_fee[kyt_provider] += KYT_FEE
if state == clean:
ckbtc_ledger.mint(utxo.value-KYT_FEE, recipient_account, uuid).await? 
checked_utxos.remove(utxo) 
else:
add_to_quarantine_list(utxo)
else:
add_to_ignore_list(utxo)
return response with UTXO statuses
</syntaxhighlight>A UTXO is considered if its value is at least the KYT fee, which is a configuration parameter of the ckBTC minter. UTXOs whose value lie below the KYT fee are added to an ignore list. The additional state <code>checked_utxos</code> is maintained to remember that a UTXO was checked if the state is clean. Once the corresponding amount of ckBTC has been minted, this state can be removed again. If the UTXO is tainted, it is moved to a quarantine list instead. UTXOs in the ignore list and quarantine list remain there.

The function <code>new_utxos</code> filters out all UTXOs on the ignore list, the quarantine list, and the sets '''''P''''' and '''''S'''''. On the other hand, the <code>checked_utxos</code> data structure is not considered.

==Converting ckBTC to BTC==

The process to convert ckBTC to BTC consists of the following steps:

#Transfer request: The user moves the ckBTC that he/she wants to convert to BTC to a special ckBTC withdrawal account under the control of the ckBTC minter and requests a transfer. The destination Bitcoin address undergoes a KYT check. If the check is successful, the request is accepted and put into a queue.
#Submission: On a heartbeat, the ckBTC minter attempts to submit transactions for validated transfer requests.
#Finalization: On a heartbeat, the ckBTC minter checks which transactions went through and finalizes these transactions.
#Resubmission: The ckBTC minter can resubmit a transaction that has been pending at least for a specific amount of time with a higher fee.

The individual parts are discussed in greater detail in the following sections.

The first step to convert ckBTC to BTC is to transfer the amount to be retrieved to the owner-specific withdrawal account under the ckBTC minter’s control. This step is required because only the ckBTC minter can burn tokens and it can only burn those tokens in one of its accounts. The withdrawal account can be obtained by calling the <code>get_withdrawal_account</code> endpoint.

After the user has transferred the desired ckBTC amount to the withdrawal account, the user can call the <code>retrieve_btc</code> endpoint to inform the ckBTC minter about the withdrawal intent. In addition to specifying the withdrawal amount, the Bitcoin address where the withdrawn funds are to be sent must be specified as well.

The ckBTC minter first performs a KYT check against the targeted Bitcoin address. If the check is successful, the ckBTC minter instructs the ckBTC ledger to burn the ckBTC in the withdrawal account. Lastly, the ckBTC minter deducts the KYT fee from the amount to be retrieved and puts the corresponding retrieval request into a queue and checks the status of the queue on a heartbeat.

If the oldest request has been in the queue for at least 10 minutes or at least 20 retrieval requests have been accumulated at the time of the heartbeat, the ckBTC minter creates a single Bitcoin transaction to serve up to 100 retrieval requests as follows:

#It selects available UTXOs with a total sum of at least the sum in the retrieval requests.
#It constructs a Bitcoin transaction with the selected UTXOs as inputs and an output for each retrieval request plus an additional output for the ckBTC minter’s fee.
# It uses the Bitcoin canister’s fee API to determine an appropriate fee for the transaction, using the median fee rate.
#It distributes the fee evenly among all outputs other than the output for the ckBTC minter’s fee.
#For each input of the transaction, the ckBTC minter invokes the threshold ECDSA functionality (calling the <code>sign_with_ecdsa</code> function) to obtain the required signatures and puts them into the transaction.
#Lastly, it sends the Bitcoin transaction by invoking the <code>send_transaction</code> function of the Bitcoin integration API.

The BTC retrieval process is depicted in the following figure.
[[File:CkBTC Specification - Withdrawal (simplified)- Version 2.png|alt=|center|thumb|600x600px|Process of converting ckBTC to BTC.]]

Note that the amounts in the transfer to the withdrawal account and the retrieval request need not be the same. The <code>retrieve_btc_status</code> endpoint can be used to query the current status of a retrieval request.

===Technical Details===

====Transfer Request====
The ckBTC minter can only burn ckBTC under its control. Therefore, the first step is to transfer ckBTC to the ckBTC withdrawal account controlled by the ckBTC minter and associated with the owner of the ckBTC that are to be burned. Specifically, the withdrawal account is a ckBTC ledger account where the principal ID is the ckBTC minter and the subaccount is derived deterministically from the owner’s principal ID.

After this transfer, the user sends a <code>retrieve_btc</code> request, specifying the amount that they would like to retrieve, the Bitcoin address where the funds are to be transferred, and, optionally, a fee level (which have yet to be defined).

Upon receiving such a request, the ckBTC minter first performs a KYT check against the Bitcoin address where funds are supposed to be sent. If the check is successful, the ckBTC minter issues a burn request to the ckBTC ledger, burning the amount in the received request in the user’s ckBTC withdrawal account. If the burn operation is successful, the ckBTC minter creates a transfer request, removing the <code>KYT_FEE</code> from the transfer amount, and appends it to its pending-requests queue.

Alternatively, if the destination address does not pass the KYT check, a burn request is sent to the ckBTC ledger, burning the <code>KYT_FEE</code> amount.

Recall that the ckBTC ledger uses the ckBTC minter default account as the minting account. Withdrawal accounts should never resolve to the default minting account because the ckBTC ledger considers the minting account for minting only.

The following pseudo-code illustrates how the <code>retrieve_btc</code> endpoint works, given the parameters <code>amount</code> and <code>btc_address</code>.<syntaxhighlight lang="rust">
assert(amount >= max(MIN_RETRIEVAL_AMOUNT, KYT_FEE))
assert(ckbtc_ledger.balance_of(withdrawal_account).await? >= amount)
(uuid, state, kyt_provider) = kyt_canister.check(btc_address).await? 

if state == clean:
index = ckbtc_ledger.burn(amount, withdrawal_account).await?
owed_kyt_fee[kyt_provider] += KYT_FEE
create_request(amount-KYT_FEE, index, btc_address, uuid)
else:
index = ckbtc_ledger.burn(KYT_FEE, withdrawal_account).await? 
owed_kyt_fee[kyt_provider] += KYT_FEE
return Error("Failed KYT check", index)
</syntaxhighlight>Note that a failure to perform the KYT check results in a rejected request. Further note that if the KYT check succeeds but the burn transaction fails (regardless of the state of the KYT response), no KYT fee is charged and the request is rejected. A subsequent request with the same parameters will result in another call to the KYT canister again, which is acceptable because multiple identical calls to the KYT service provider do not result in increased fees.

For each retrieval request, the ckBTC minter stores the following data:

*<code>block_index</code>: The block index of the burn operation used to burn the ckBTC. Since the block index is unique, it is used as the request ID.
*<code>amount</code>: The total amount of tokens to retrieve. This amount must be at least the minimum retrieval amount as defined above.
*<code>address</code>: The address where the bitcoins will be sent.
*<code>received_at</code>: The timestamp when the request was received.
*<code>fee_level</code>: If no fee level is provided, a medium fee level is used.

====Submission====
The ckBTC minter uses the heartbeat mechanism to initiate Bitcoin transfers. On a heartbeat, the following steps are carried out:

#Check if there is at least one request that is 10 minutes old or there are at least 20 requests in the pending-requests queue. If not, stop.
#Update the balance of the ckBTC minter’s main BTC address (the P2WPKH address derived from its public key with an empty derivation path) using the Bitcoin integration’s <code>get_utxos</code> function.
#Determine the total amount of bitcoins available, which is the sum of all bitcoins in processed UTXOs, plus bitcoins under the ckBTC minter’s main BTC address.
#Call the transfer function with the next batch of requests with the same fee level that can be served given the total amount of available bitcoins. A transaction is created, setting the transaction ID for each request in the batch, and sent to the Bitcoin network.
#Every request in this batch is then moved to the unconfirmed-transfers queue.

As evident from the steps outlined above, the transfer function can handle multiple requests at the same time. Handling multiple requests in a single transaction has several advantages over sending individual transactions:

#Requests can possibly be served more quickly, especially if the ckBTC minter must wait for change to return to its main BTC address.
#As the fee for the non-input bytes is shared, the fee per request is slightly lower.
#Serving multiple requests at the same time makes denial-of-service attacks where an attacker attempts to keep the pool of usable UTXOs empty with many small requests harder.

Given this set of requests, the next step is to select UTXOs for the transaction.

Since UTXOs are always spent entirely, the difference between the sum of bitcoins in the spent UTXOs and the requested amount must be transferred to a new UTXO as well. The ckBTC minter uses its main BTC address to accumulate “change”.

All UTXOs of its main BTC address are immediately transferred to the corresponding set '''''P''''' of processed UTXOs. The reason is that there is no minting for the balances in these UTXOs.

The transfer function performs the following steps:

#First, the transfer function determines the “target”, the total number t (≤ T) of bitcoins that must be transferred out to handle all requests in the given batch.
# Then, the function selects UTXOs for the transaction from one or more sets '''''P''''' of its managed addresses. The selected UTXOs are moved from the sets '''''P''''' to the corresponding sets '''''S'''''.
# Next, the Bitcoin transaction is built and computes the fee based on current Bitcoin fees (using the <code>get_current_fee_percentiles</code> Bitcoin integration endpoint) and the size of the transaction. The fee rate in Satoshi/vbyte is determined by the fee level. Note that the fee is deducted by splitting it evenly among the handled retrieval requests, deducting the same fraction of the total fee from each output that is not returning change and the ckBTC minter fee to the ckBTC minter. The ckBTC minter fee is <code>246*in + 7*out + 52 Satoshi</code>, where in and out denote the number of transaction inputs and outputs, respectively (as specified here).
#Every input is signed using the threshold ECDSA interface.
#Finally, the transaction is submitted using the <code>send_transaction</code> Bitcoin integration endpoint.

The following UTXO selection algorithm, in pseudo-code, is used:<syntaxhighlight lang="rust">
// t = target (satoshis), P = Union of all sets of processed UTXOs
// Pre-condition: sum(P) >= t
select_utxos(t, P):
return greedy(t, P)

greedy(t, A): // t = target, A = Available UTXOs
if t ≤ 0 or |A| = 0: return {}
m := max(A) // The UTXO with the largest value
if m.value < t:
return {m} ∪ greedy(t-m.value, A \ {m})
else:
return min({p ∊ A | p.value ≥ t})

</syntaxhighlight>The algorithm has the following properties:

*It uses the smallest number of UTXOs possible for the given target.
*If a single UTXO suffices, it uses the UTXO that results in the smallest change.

Once the transaction is sent, the requests are moved to the unconfirmed-transfers queue.

====Finalization====
The ckBTC minter uses the heartbeat mechanism to determine the status of sent transactions as well. Specifically, the ckBTC minter periodically wakes up and checks the state of the requests in the unconfirmed-transfers queue.The ckBTC minter checks the UTXOs of its main account to determine which transactions have sufficiently many confirmations.

Such requests can then be stored in a confirmed-transfers queue for some time in order to enable a more long-term look-up of the transaction status.

====Resubmission====
It is possible that it takes a long time for a transaction to be included in a block. If fees increase significantly for some time, a transaction may even be stuck for a long time or dropped entirely. While the ckBTC minter enforces a reasonable fee through its fee levels, it may still be necessary to issue a transaction again because burned ckBTC are never returned and UTXOs are never freed and are thus stuck when the transaction spending these UTXOs is stuck.

The ckBTC minter resubmits a transaction that has not been confirmed within a certain number of hours. Given that the ckBTC minter always waits for many confirmations, the waiting time will likely be multiple hours up to one day, though this parameter has not been defined yet.

The new transaction will use the UTXOs reserved for the requests. The transaction will be identical except that the outputs for each user will be reduced due to the increased fee. The new fee is the sum of the old transaction fee plus the size of the transaction (in vbytes) times the minimum relay fee of 1 Satoshi/vbyte plus the ckBTC minter fee again because it must acquire new signatures and send the new transaction to the Bitcoin canister.

==Fees==

The ckBTC canisters run on an application subnet and must be self-sustainable. Rather than charging cycles for the endpoints, the ckBTC minter accumulates a surplus of BTC over time. In the future, the ckBTC minter will mint ckBTC to get the total ckBTC supply and the BTC amount under the ckBTC minter's control to match. The ckBTC minter can then trade these extra ckBTC tokens for cycles to charge both the ckBTC minter and ckBTC ledger.

There is a growing surplus of BTC because there is a ckBTC transfer fee of 0.0000001 ckBTC, which is burned, and a ckBTC minter fee when retrieving BTC. The latter is determined as follows:

*Under the conservative assumption that 1 BTC = 10,000 XDR, 1 billion cycles corresponds to 10 Satoshi (because 1 trillion cycles corresponds to 1 XDR).
* The [https://internetcomputer.org/docs/current/developer-docs/deploy/computation-and-storage-costs/ cost] to obtain a single EDCSA signature is approximately 21.54 billion cycles on a 28-node subnet, whereas sending a Bitcoin transaction costs 5 billion cycles plus 20 million cycles per byte.

Given these numbers, the cost to sign and send a transaction with <code>in</code> inputs and <code>out</code> outputs is

21.54b*in + 5b + tx_size*20m cycles
< 21.54b*in + 5b + (149*in + 35*out + 10)*20m cycles
< 24.52b*in +0.7b*out + 5.2b cycles
< 246*in + 7*out + 52 satoshi.

The formula <code>246*in + 7*out + 52</code> is used to determine the ckBTC minter’s fee in satoshi. Since every transaction has at least one input and one output, the fee is at least 305 Satoshi.

This conservative pricing strategy is used to subsidize the other endpoints, which are free of charge. Moreover, while the <code>retrieve_btc</code> endpoint is relatively expensive, the fee is typically still lower than the Bitcoin miner fee.

As mentioned above, there is also a KYT fee (currently 2000 Satoshi) when converting BTC to ckBTC and vice versa.

==ckBTC Minter API==

The ckBTC minter provides the following endpoints:

*<code>get_btc_address</code>: Returns a specific Bitcoin address that the caller can use to obtain ckBTC by sending BTC to this address.
*<code>update_balance</code>: Instructs the ckBTC minter to check the balance of a Bitcoin address and mint ckBTC into the account of the owner.
*<code>estimate_withdrawal_fee</code>: Returns a current estimate for the fee to be paid when retrieving a certain BTC amount.
* <code>get_deposit_fee</code>: Returns the fee charged when minting ckBTC.
*<code>get_withdrawal_account</code>: Returns a specific ckBTC account where the owner must transfer ckBTC before being able to retrieve BTC.
*<code>retrieve_btc</code>: Instructs the ckBTC minter to burn a certain ckBTC amount and send the corresponding BTC amount, minus fees, to a provided Bitcoin address.
* <code>retrieve_btc_status</code>: Returns the status of a previous <code>retrieve_btc</code> call.
*<code>get_minter_info</code>: Returns information about the ckBTC minter itself.
*<code>get_events</code>: Returns a set of events at the ckBTC minter.

The endpoints are discussed in more detail in the [https://internetcomputer.org/docs/current/developer-docs/integrations/bitcoin/ckbtc ckBTC developer documentation.]

== See also==

*Code Bitcoin on the Internet Computer: https://internetcomputer.org/bitcoin-integration
* Chain-key tokens: https://internetcomputer.org/how-it-works/chain-key-tokens/
*Chain-key Bitcoin developer documents: https://internetcomputer.org/docs/current/developer-docs/integrations/bitcoin/ckbtc

Chain-key Bitcoin

2023-07-17T17:19:34Z

Jamongeon: Editing and adding information from the ckBTC design document (https://docs.google.com/document/d/18n33mTyvZaaisN8KU4FeliITbrL8zrUCKxv6tVfXPH0/edit?pli=1#heading=h.kknwh0c6ltfe) as requested and discussed with @Thomas Locher

== Overview ==

Chain-key Bitcoin (ckBTC) is a token on the Internet Computer that is backed 1:1 by Bitcoin (BTC) such that 1 ckBTC can always be redeemed for 1 BTC and vice versa.

Unlike other tokens pegged to Bitcoin, the ckBTC token does not rely on a third-party bridge for the conversion between BTC and ckBTC, making it a substantially more secure alternative to “wrapped” tokens.

While chain-key Bitcoin and regular Bitcoin have the same value, the advantage of chain-key Bitcoin is fast and cheap transfers: A transfer is finalized within a few seconds (a speed-up of roughly three orders of magnitude compared to transfers on the Bitcoin blockchain) and only costs 0.0000001 ckBTC (approximately two orders of magnitude lower than the Bitcoin miner fees).

== Architecture ==

The ckBTC functionality is built upon the [[Bitcoin integration]] of the Internet Computer, which makes it possible for canisters to receive, hold, and send bitcoin.

There are two canisters, the ckBTC minter and ckBTC ledger, that together provide the ckBTC functionality. The ckBTC minter mints new ckBTC tokens whenever it receives Bitcoin. Likewise, it burns ckBTC tokens whenever an owner of ckBTC tokens requests a withdrawal of Bitcoin. The ckBTC minter needs to receive BTC, based on a large number of confirmations due to the lack of finality in Bitcoin, before it mints the same amount in ckBTC and it burns ckBTC before it transfers BTC back to the users.

A best-effort approach is used for the transfer of BTC to the users, i.e., the user cannot get the burned ckBTC back. Rather, the ckBTC minter repeatedly attempts to transfer BTC out. Note that “user” refers to the caller of the functions exposed by the ckBTC minter and ckBTC ledger in the following. A human user or another canister may be behind the function invocations.

The ckBTC ledger is [https://github.com/dfinity/ICRC-1/blob/main/standards/ICRC-1/README.md ICRC-1 compliant], updating the balance accounts when ckBTC tokens are transferred and executing the mint and burn operations coming from the ckBTC minter.

An overview of the basic architecture is depicted in the following figure.
[[File:CkBTC Specification - Overview (Version 2).png|alt=|center|thumb|600x600px|High-level overview of chain-key Bitcoin.]]
The figure shows the main flow at a high level of abstraction: Users interact with the ckBTC minter and the ckBTC ledger to convert ckBTC/BTC and transfer ckBTC, respectively. The ckBTC minter interacts with the Bitcoin canister to retrieve information about the Bitcoin network state and send Bitcoin transactions.

The ckBTC minter further interacts with the [[Know-Your-Transaction (KYT) Canister|KYT canister]] for "know-your-transaction" (KYT) checks. These checks are meant to ensure that the ckBTC minter only uses "clean" Bitcoins to back the issued ckBTC tokens and to prevent transferring Bitcoins to Bitcoin addresses that are considered to be associated with illicit activity. As such, these KYT checks provide an additional layer of security to ckBTC users.

== Technical Details ==

=== Bitcoin Addresses ===
Within this page, Bitcoin addresses are implicitly understood to be of type P2WPKH (“pay to witness public key hash”) as defined in [https://en.bitcoin.it/wiki/BIP_0141 BIP-141]. All addresses are rendered in the Bech32 format as defined in [https://github.com/bitcoin/bips/blob/master/bip-0173.mediawiki BIP-173]. The main advantage of P2WPKH over the legacy P2PKH address type is that its use results in lower transaction fees.

For this reason, it is the default address type in most wallets for standard transactions.

=== ckBTC Ledger ===
The ckBTC ledger is a [https://dashboard.internetcomputer.org/canister/mxzaz-hqaaa-aaaar-qaada-cai canister], controlled by the NNS (specifically, the [https://dashboard.internetcomputer.org/canister/r7inp-6aaaa-aaaaa-aaabq-cai NNS root canister]), running on the [https://dashboard.internetcomputer.org/subnet/pzp6e-ekpqk-3c5x7-2h6so-njoeq-mt45d-h3h6c-q3mxf-vpeq5-fk5o7-yae pzp6e] subnet. Since it is ICRC-1 compliant, it offers [https://github.com/dfinity/ICRC-1/blob/aa82e52aaa74cc7c5f6a141e30b708bf42ede1e3/standards/ICRC-1/README.md#methods all functions] defined in the ICRC-1 standard. In particular, it provides the functionality to transfer ckBTC between accounts.

The ckBTC ledger is responsible for keeping account balances and for transferring ckBTC between accounts. It provides the following functionality:

* It enables the ckBTC minter to mint and burn ckBTC.
* It enables the transfer of ckBTC among users.

As mentioned above, the transfer fee is 0.0000001 ckBTC, the equivalent of 10 Satoshi. Since the fee in ckBTC is burned, every transaction leads to a growing surplus in BTC controlled by the ckBTC minter. The surplus will eventually be used to cover the cycle cost of the ckBTC minter.

The minting account is the ckBTC minter’s default account; that is, the account derived from the ckBTC minter’s principal ID and the all-zero subaccount.

The initial supply of the ckBTC ledger is 0. ckBTC tokens are minted only when the ckBTC minter receives Bitcoin, ensuring that the ckBTC supply managed by the ckBTC ledger is upper bounded by the amount of Bitcoin held by the ckBTC minter.

=== ckBTC Minter ===

The ckBTC minter is a [https://dashboard.internetcomputer.org/canister/mqygn-kiaaa-aaaar-qaadq-cai canister] that is controlled by the NNS and running on the [https://dashboard.internetcomputer.org/subnet/pzp6e-ekpqk-3c5x7-2h6so-njoeq-mt45d-h3h6c-q3mxf-vpeq5-fk5o7-yae pzp6e] subnet as well.

The ckBTC minter is the canister responsible for managing deposited BTC and minting/burning ckBTC based on the amount of deposited BTC. It provides the following functionality:

* For a certain principal ID and an optional subaccount, it returns a specific Bitcoin address under the ckBTC minter’s control.
* Users can inform the ckBTC minter about Bitcoins that were sent to an address controlled by the ckBTC minter. If the balance has increased, the ckBTC minter mints the same amount in ckBTC for the user associated with the Bitcoin address.
* Users can ask the ckBTC minter to track the balance of a Bitcoin address until new funds are received, which then automatically triggers the minting of the equivalent amount of ckBTC.
* Users can request to get bitcoins back. The ckBTC minter burns the same amount of ckBTC and transfers the Bitcoins to the address provided by the user.

The ckBTC minter canister has a few important configuration parameters including:

* <code>retrieve_btc_min_amount</code>: This is the minimum ckBTC amount that can be burned and, correspondingly, the minimum BTC amount that can be withdrawn. The parameter is set to 0.001 BTC, or 100,000 satoshi.
* <code>max_time_in_queue_nanos</code>: Any BTC retrieval request should be kept in a queue for at most this time. Caching requests rather than handling them right away has the advantage that multiple requests can be served in a single transaction, saving Bitcoin miner fees. The parameter is set to 10 minutes, which corresponds to the expected time between Bitcoin blocks.
* <code>min_confirmations</code>: The number of confirmations required for the ckBTC minter to accept a Bitcoin transaction. In particular, the ckBTC minter does not mint ckBTC before a transaction transferring BTC to a Bitcoin address managed by the ckBTC minter reaches this number of transactions. The parameter was initially set to 72 but has been reduced to 12 in the meantime.
* <code>kyt_fee</code>: The fee that must be paid for KYT checks. It is currently set to 2000 satoshi.
The other parameters are self-explanatory and can be found in the [https://github.com/dfinity/ic/blob/master/rs/bitcoin/ckbtc/minter/ckbtc_minter.did ckBTC minter Candid file].

=== State Management ===

==== Managed Addresses ====
A Bitcoin address that is controlled by the ckBTC minter and has a positive balance is called a managed address. If the balance of a managed address reduces to zero, the address is no longer managed and can be removed from the state.

Note that it is possible to increase the balance again, in which case the address becomes managed again.

==== Unspent Transaction Outputs ====
Once a new Unspent Transaction Output (UTXO) under the control of the ckBTC minter is discovered (using the <code>update_balance</code> function), it is stored internally in a set called <code>available_utxos</code> (defined [https://github.com/dfinity/ic/blob/2348b094d3d27616ee3f049d3048baa1da8d625a/rs/bitcoin/ckbtc/minter/src/state.rs#L305C14-L305C14 here] in the source code).

All discovered UTXOs remain in this set until a Bitcoin transaction is created to spend one or more of them when retrieving Bitcoins. When a transaction is created spending some UTXOs, these UTXOs are removed from the set <code>available_utxos</code> and inserted in the <code>used_utxos</code> field of the <code>SubmittedBtcTransaction</code> struct (defined [https://github.com/dfinity/ic/blob/70d19f16c17f8f42987a46d473ba27705927cdb7/rs/bitcoin/ckbtc/minter/src/state.rs#L87 here] in the source code), which is the internal representation of a Bitcoin transaction.

A UTXO is removed from the ckBTC state when the <code>SubmittedBtcTransaction</code> struct that contains the UTXO is removed from the state.

==== Transactions ====
To provide information about current transactions, outgoing transactions must be cached. A transaction is cached until its inputs are no longer considered unspent based on a large number of confirmations.

Every transaction that the ckBTC minter creates has an output that sends the ckBTC minter fee plus the transaction change back to its main BTC address (P2WPKH address derived from its public key with an empty derivation path) using the Bitcoin integration’s <code>get_utxos</code> function.

A transaction can be removed from the cache if the transaction output that belongs to the ckBTC minter appears in the returned list when calling <code>get_utxos</code> with sufficient confirmations for the ckBTC minter’s main BTC address.

The ckBTC minter may resubmit transactions, making use of Bitcoin’s request by fee (RBF) mechanism as defined in [https://github.com/bitcoin/bips/blob/master/bip-0125.mediawiki BIP-125]. In the case of ckBTC, a resubmission adds a transaction to the cache that spends exactly the same UTXOs as the transaction it replaces. The only difference is that the BTC amount sent to the user is reduced in order to increase the fee.

BIP-125 states that at most 100 transactions may be evicted from the mempool, i.e., the fee cannot be increased more than 100 times. Moreover, the fee must be increased at least by the minimum relay fee (see minrelaytxfee [https://en.bitcoin.it/wiki/Miner_fees#Relaying here]) of 1 Satoshi/vbyte.

For example, if we assume a minimum increase of 200 Satoshi (the minimum fee for a basic segwit transaction with one input and one output is 192 Satoshi and the number per output is always lower than 200 if there are at least as many outputs as inputs), the minimum transfer amount should be at least 20,000 Satoshi which equals 0.0002 BTC. When adding a base fee at a large fee rate of 100 Satoshi/vbyte and assuming a virtual transaction size of 200 vbyte per output, we get a minimum transfer amount of 0.0004 BTC. Adding a security margin, we define the minimum retrieval amount to be <code>MIN_RETRIEVAL_AMOUNT = 0.001 BTC</code>.

''To ensure that every transaction can potentially be updated, the RBF flag must be set on every transaction.''

==== Know Your Transaction & Fees ====
Before UTXOs are accepted, they undergo a know-your-transaction (KYT) check where information about the UTXO is sent to a “KYT canister”, which interacts with KYT service providers, such as Chainalysis, to check if the UTXO is “clean” using HTTPS outcalls.

In a similar fashion, there is a KYT check of the destination address when attempting to transfer Bitcoins out of the ckBTC minter.

Both of these KYT checks incur fees, charged in ckBTC:

* The KYT fee is subtracted from the amount to be minted and the amount transferred out.
* The KYT fee is 2000 satoshi per KYT check.

The KYT canister supports multiple KYT access key providers, called maintainers, each having a principal ID and (secret) access key to the external KYT service. Since the maintainers pay for their subscriptions, they must be remunerated. To this end, the ckBTC minter maintains a mapping of maintainers to the owed amount, which is the number of KYT checks that were performed with the respective maintainer’s credentials times the KYT fee of 2000 Satoshi.

Each successful response from the KYT canister contains the principal ID of the maintainer so that the ckBTC minter can update the owed amount correctly. Once every 24 hours, the ckBTC minter pays out the owed amounts in ckBTC to all maintainers.

== Converting BTC to ckBTC ==

In this section, the process to convert BTC to ckBTC is explained, making use of the ckBTC minter and ckBTC ledger endpoints.

The first step is for the user to determine the Bitcoin address where the user is supposed to transfer Bitcoin for the minting process by calling the <code>get_btc_address</code> endpoint. Next, the user transfers the desired BTC amount to this Bitcoin address. Subsequently, the user calls the <code>track_balance</code> endpoint to inform the ckBTC minter that it is about to receive Bitcoins at the given address.

Internally, the tracking works as follows. Since the expected time between Bitcoin blocks is 10 minutes, the ckBTC minter first checks if new unspent transaction outputs (UTXOs) are available for that address with at least the required number of confirmations after <code>10*min_confirmations</code> minutes. If there is at least one new UTXO, tracking stops and the ckBTC minter instructs the ckBTC ledger to mint the same amount of ckBTC tokens into the account derived from the principal ID and the subaccount.

If no new UTXOs are discovered with sufficiently many confirmations, the ckBTC ledger checks if there are new UTXOs with at least one confirmation. If this is not the case, tracking stops as well. Otherwise, the expected time until the first new UTXO reaches the desired number of confirmations is computed, which is 10 minutes times the difference between the desired number of confirmations and the current number of confirmations. The same process then repeats until ckBTC tokens are minted or tracking stops.

It is evident from this description that it’s possible that tracking may stop before ckBTC tokens are minted, for example, if it takes an unusually long time until the transaction appears in a block. In this case, the endpoint can be invoked again. Alternatively, the user can wait until the transaction has the required number of confirmations and call the <code>update_balance</code> endpoint.

The following figure depicts the conversion flow using the <code>track_balance</code> endpoint.
[[File:CkBTC Specification - Submission (simplified) - Version 2.png|alt=|center|thumb|600x600px|Process of converting BTC to ckBTC.]]

In this figure, the ckBTC minter discovers the new UTXO after the waiting period by calling <code>get_utxos</code> on the Bitcoin canister. As mentioned above, it is possible that it may take several interactions with the Bitcoin canister until the transaction has received sufficiently many confirmations before the ckBTC minter issues the mint transaction to the ckBTC ledger. After a successful KYT check, the ckBTC ledger is instructed to mint ckBTC tokens for the user. The minted amount corresponds to the value of the UTXO minus the KYT fee.

=== Technical Details ===
As mentioned above, a call to the <code>update_balance</code> endpoint triggers a call to the Bitcoin canister to get UTXOs for the Bitcoin address associated with the given <code>principal-subaccount</code> pair. Let '''''R''''' denote the set of returned UTXOs. The following pseudo-code illustrates how the UTXOs are processed:<syntaxhighlight lang="rust">
for utxo in new_utxos(R): // R = set of received UTXOs in the get_utxos call
if utxo.value >= KYT_FEE:
if utxo in checked_utxos: 
(uuid, state) = checked_utxos.get(utxo)
else:
(uuid, state, kyt_provider) = kyt_canister.check(utxo).await?
if state == clean:
D = D \ {utxo}
P = P ∪ {utxo}
checked_utxos.set(utxo, (uuid, clean))
owed_kyt_fee[kyt_provider] += KYT_FEE
if state == clean:
ckbtc_ledger.mint(utxo.value-KYT_FEE, recipient_account, uuid).await? 
checked_utxos.remove(utxo) 
else:
add_to_quarantine_list(utxo)
else:
add_to_ignore_list(utxo)
return response with UTXO statuses
</syntaxhighlight>A UTXO is considered if its value is at least the KYT fee, which is a configuration parameter of the ckBTC minter. UTXOs whose value lie below the KYT fee are added to an ignore list. The additional state <code>checked_utxos</code> is maintained to remember that a UTXO was checked if the state is clean. Once the corresponding amount of ckBTC has been minted, this state can be removed again. If the UTXO is tainted, it is moved to a quarantine list instead. UTXOs in the ignore list and quarantine list remain there.

The function <code>new_utxos</code> filters out all UTXOs on the ignore list, the quarantine list, and the sets '''''P''''' and '''''S'''''. On the other hand, the <code>checked_utxos</code> data structure is not considered.

=== Automated Tracking ===
The <code>update_balance</code> function only triggers a minting operation if the corresponding Bitcoin transaction has sufficient confirmations, which implies that it is the user’s responsibility to check the number of confirmations before making the <code>update_balance</code> call.

In order to improve the user experience, the additional endpoint <code>track_balance</code>, with the same parameters as <code>update_balance</code>, is provided. When invoked instead of <code>update_balance</code>, the ckBTC minter starts tracking the Bitcoin address derived from the provided principal ID and subaccount using the <code>get_btc_address</code> endpoint. If no principal ID is provided, then the sender’s principal ID is used. If no subaccount is provided, then the default subaccount (all zeros) is used.

The balance of the Bitcoin address is not tracked indefinitely. Tracking is stopped if either at least one new unspent transaction output (UTXO) is discovered or there is no new UTXO within a certain time interval as defined in the following:

* Since the expected time between Bitcoin blocks is 10 minutes, the ckBTC minter first checks if new UTXOs are available for that address with at least the required number of confirmations after <code>10*MIN_CONFIRMATIONS</code> minutes.
* If there is at least one new UTXO, tracking stops and the same flow as for <code>update_balance</code> is triggered. If the KYT check is successful, the ckBTC minter instructs the ckBTC ledger to mint the same amount of ckBTC tokens minus the KYT fee into the account derived from the principal ID and the subaccount.
* If no new UTXOs are discovered with sufficiently many confirmations, the ckBTC ledger checks if there are new UTXOs with at least one confirmation. If this is not the case, tracking stops as well. Otherwise, the expected time until the first new UTXO reaches the desired number of confirmations is computed, which is 10 minutes times the difference between the desired number of confirmations and the current number of confirmations. The next check is scheduled after this many minutes and the process repeats until a new UTXO is discovered or tracking stops.

It is evident from this description that it is possible that tracking may stop before ckBTC tokens are minted, for example, if it takes an unusually long time until the transaction appears in a block. In this case, the endpoint can be invoked again. Alternatively, the user can wait until the transaction has the required number of confirmations and call the <code>update_balance</code> endpoint.

== Converting ckBTC to BTC ==

The process to convert ckBTC to BTC consists of the following steps:

# Transfer request: The user moves the ckBTC that he/she wants to convert to BTC to a special ckBTC withdrawal account under the control of the ckBTC minter and requests a transfer. The destination Bitcoin address undergoes a KYT check. If the check is successful, the request is accepted and put into a queue.
# Submission: On a heartbeat, the ckBTC minter attempts to submit transactions for validated transfer requests.
# Finalization: On a heartbeat, the ckBTC minter checks which transactions went through and finalizes these transactions.
# Resubmission: The ckBTC minter can resubmit a transaction that has been pending at least for a specific amount of time with a higher fee.

The individual parts are discussed in greater detail in the following sections.

The first step to convert ckBTC to BTC is to transfer the amount to be retrieved to the owner-specific withdrawal account under the ckBTC minter’s control. This step is required because only the ckBTC minter can burn tokens and it can only burn those tokens in one of its accounts. The withdrawal account can be obtained by calling the <code>get_withdrawal_account</code> endpoint.

After the user has transferred the desired ckBTC amount to the withdrawal account, the user can call the <code>retrieve_btc</code> endpoint to inform the ckBTC minter about the withdrawal intent. In addition to specifying the withdrawal amount, the Bitcoin address where the withdrawn funds are to be sent must be specified as well.

The ckBTC minter first performs a KYT check against the targeted Bitcoin address. If the check is successful, the ckBTC minter instructs the ckBTC ledger to burn the ckBTC in the withdrawal account. Lastly, the ckBTC minter deducts the KYT fee from the amount to be retrieved and puts the corresponding retrieval request into a queue and checks the status of the queue on a heartbeat.

If the oldest request has been in the queue for at least 10 minutes or at least 20 retrieval requests have been accumulated at the time of the heartbeat, the ckBTC minter creates a single Bitcoin transaction to serve up to 100 retrieval requests as follows:

# It selects available UTXOs with a total sum of at least the sum in the retrieval requests.
# It constructs a Bitcoin transaction with the selected UTXOs as inputs and an output for each retrieval request plus an additional output for the ckBTC minter’s fee.
# It uses the Bitcoin canister’s fee API to determine an appropriate fee for the transaction, using the median fee rate.
# It distributes the fee evenly among all outputs other than the output for the ckBTC minter’s fee.
# For each input of the transaction, the ckBTC minter invokes the threshold ECDSA functionality (calling the <code>sign_with_ecdsa</code> function) to obtain the required signatures and puts them into the transaction.
# Lastly, it sends the Bitcoin transaction by invoking the <code>send_transaction</code> function of the Bitcoin integration API.

The BTC retrieval process is depicted in the following figure.
[[File:CkBTC Specification - Withdrawal (simplified)- Version 2.png|alt=|center|thumb|600x600px|Process of converting ckBTC to BTC.]]

Note that the amounts in the transfer to the withdrawal account and the retrieval request need not be the same. The <code>retrieve_btc_status</code> endpoint can be used to query the current status of a retrieval request.

=== Technical Details ===

==== Transfer Request ====
The ckBTC minter can only burn ckBTC under its control. Therefore, the first step is to transfer ckBTC to the ckBTC withdrawal account controlled by the ckBTC minter and associated with the owner of the ckBTC that are to be burned. Specifically, the withdrawal account is a ckBTC ledger account where the principal ID is the ckBTC minter and the subaccount is derived deterministically from the owner’s principal ID.

After this transfer, the user sends a <code>retrieve_btc</code> request, specifying the amount that they would like to retrieve, the Bitcoin address where the funds are to be transferred, and, optionally, a fee level (which have yet to be defined).

Upon receiving such a request, the ckBTC minter first performs a KYT check against the Bitcoin address where funds are supposed to be sent. If the check is successful, the ckBTC minter issues a burn request to the ckBTC ledger, burning the amount in the received request in the user’s ckBTC withdrawal account. If the burn operation is successful, the ckBTC minter creates a transfer request, removing the <code>KYT_FEE</code> from the transfer amount, and appends it to its pending-requests queue.

Alternatively, if the destination address does not pass the KYT check, a burn request is sent to the ckBTC ledger, burning the <code>KYT_FEE</code> amount.

The steps of a successful request are illustrated in the following figure.

Recall that the ckBTC ledger uses the ckBTC minter default account as the minting account. Withdrawal accounts should never resolve to the default minting account because the ckBTC ledger considers the minting account for minting only.

The following pseudo-code illustrates how the <code>retrieve_btc</code> endpoint works, given the parameters <code>amount</code> and <code>btc_address</code>.<syntaxhighlight lang="rust">
assert(amount >= max(MIN_RETRIEVAL_AMOUNT, KYT_FEE))
assert(ckbtc_ledger.balance_of(withdrawal_account).await? >= amount)
(uuid, state, kyt_provider) = kyt_canister.check(btc_address).await? 

if state == clean:
index = ckbtc_ledger.burn(amount, withdrawal_account).await?
owed_kyt_fee[kyt_provider] += KYT_FEE
create_request(amount-KYT_FEE, index, btc_address, uuid)
else:
index = ckbtc_ledger.burn(KYT_FEE, withdrawal_account).await? 
owed_kyt_fee[kyt_provider] += KYT_FEE
return Error("Failed KYT check", index)
</syntaxhighlight>Note that a failure to perform the KYT check results in a rejected request. Further note that if the KYT check succeeds but the burn transaction fails (regardless of the state of the KYT response), no KYT fee is charged and the request is rejected. A subsequent request with the same parameters will result in another call to the KYT canister again, which is acceptable because multiple identical calls to the KYT service provider do not result in increased fees.

For each retrieval request, the ckBTC minter stores the following data:

* <code>block_index</code>: The block index of the burn operation used to burn the ckBTC. Since the block index is unique, it is used as the request ID.
* <code>amount</code>: The total amount of tokens to retrieve. This amount must be at least the minimum retrieval amount as defined above.
* <code>address</code>: The address where the Bitcoins will be sent.
* <code>received_at</code>: The timestamp when the request was received.
* <code>fee_level</code>: If no fee level is provided, a medium fee level is used.

==== Submission ====
The ckBTC minter uses the heartbeat mechanism to initiate Bitcoin transfers. On a heartbeat, the following steps are carried out:

# Check if there is at least one request that is 10 minutes old or there are at least 20 requests in the pending-requests queue. If not, stop.
# Update the balance of the ckBTC minter’s main BTC address (the P2WPKH address derived from its public key with an empty derivation path) using the Bitcoin integration’s <code>get_utxos</code> function.
# Determine the total amount of Bitcoins available, which is the sum of all Bitcoins in processed UTXOs, plus Bitcoins under the ckBTC minter’s main BTC address.
# Call the transfer function with the next batch of requests with the same fee level that can be served given the total amount of available Bitcoins. A transaction is created, setting the transaction ID for each request in the batch, and sent to the Bitcoin network.
# Every request in this batch is then moved to the unconfirmed-transfers queue.

As evident from the steps outlined above, the transfer function can handle multiple requests at the same time. Handling multiple requests in a single transaction has several advantages over sending individual transactions:

# Requests can possibly be served more quickly, especially if the ckBTC minter must wait for change to return to its main BTC address.
# As the fee for the non-input bytes is shared, the fee per request is slightly lower.
# Serving multiple requests at the same time makes denial-of-service attacks where an attacker attempts to keep the pool of usable UTXOs empty with many small requests harder.

Given this set of requests, the next step is to select UTXOs for the transaction.

Since UTXOs are always spent entirely, the difference between the sum of Bitcoins in the spent UTXOs and the requested amount must be transferred to a new UTXO as well. The ckBTC minter uses its main BTC address to accumulate “change”.

All UTXOs of its main BTC address are immediately transferred to the corresponding set '''''P''''' of processed UTXOs. The reason is that there is no minting for the balances in these UTXOs.

The transfer function performs the following steps:

# First, the transfer function determines the “target”, the total number t (≤ T) of Bitcoins that must be transferred out to handle all requests in the given batch.
# Then, the function selects UTXOs for the transaction from one or more sets '''''P''''' of its managed addresses. The selected UTXOs are moved from the sets '''''P''''' to the corresponding sets '''''S'''''.
# Next, the Bitcoin transaction is built and computes the fee based on current Bitcoin fees (using the <code>get_current_fee_percentiles</code> Bitcoin integration endpoint) and the size of the transaction. The fee rate in Satoshi/vbyte is determined by the fee level. Note that the fee is deducted by splitting it evenly among the handled retrieval requests, deducting the same fraction of the total fee from each output that is not returning change and the ckBTC minter fee to the ckBTC minter. The ckBTC minter fee is <code>246*in + 7*out + 52 Satoshi</code>, where in and out denote the number of transaction inputs and outputs, respectively (as specified here).
# Every input is signed using the threshold ECDSA interface.
# Finally, the transaction is submitted using the <code>send_transaction</code> Bitcoin integration endpoint.

The following UTXO selection algorithm, in pseudo-code, is used:<syntaxhighlight lang="rust">
// t = target (satoshis), P = Union of all sets of processed UTXOs
// Pre-condition: sum(P) >= t
select_utxos(t, P):
return greedy(t, P)

greedy(t, A): // t = target, A = Available UTXOs
if t ≤ 0 or |A| = 0: return {}
m := max(A) // The UTXO with the largest value
if m.value < t:
return {m} ∪ greedy(t-m.value, A \ {m})
else:
return min({p ∊ A | p.value ≥ t})

</syntaxhighlight>The algorithm has the following properties:

* It uses the smallest number of UTXOs possible for the given target.
* If a single UTXO suffices, it uses the UTXO that results in the smallest change.

Once the transaction is sent, the requests are moved to the unconfirmed-transfers queue.

==== Finalization ====
The ckBTC minter uses the heartbeat mechanism to determine the status of sent transactions as well. Specifically, the ckBTC minter periodically wakes up and checks the state of the requests in the unconfirmed-transfers queue.The ckBTC minter checks the UTXOs of its main account to determine which transactions have sufficiently many confirmations.

Such requests can then be stored in a confirmed-transfers queue for some time in order to enable a more long-term look-up of the transaction status.

==== Resubmission ====
It is possible that it takes a long time for a transaction to be included in a block. If fees increase significantly for some time, a transaction may even be stuck for a long time or dropped entirely. While the ckBTC minter enforces a reasonable fee through its fee levels, it may still be necessary to issue a transaction again because burned ckBTC are never returned and UTXOs are never freed and are thus stuck when the transaction spending these UTXOs is stuck.

The ckBTC minter resubmits a transaction that has not been confirmed within a certain number of hours. Given that the ckBTC minter always waits for many confirmations, the waiting time will likely be multiple hours up to one day, though this parameter has not been defined yet.

The new transaction will use the UTXOs reserved for the requests. The transaction will be identical except that the outputs for each user will be reduced due to the increased fee. The new fee is the sum of the old transaction fee plus the size of the transaction (in vbytes) times the minimum relay fee of 1 Satoshi/vbyte plus the ckBTC minter fee again because it must acquire new signatures and send the new transaction to the Bitcoin canister.

== Fees ==

The ckBTC canisters run on an application subnet and must be self-sustainable. Rather than charging cycles for the endpoints, the ckBTC minter accumulates a surplus of BTC over time. In the future, the ckBTC minter will mint ckBTC to get the total ckBTC supply and the BTC amount under the ckBTC minter's control to match. The ckBTC minter can then trade these extra ckBTC tokens for cycles to charge both the ckBTC minter and ckBTC ledger.

There is a growing surplus of BTC because there is a ckBTC transfer fee of 0.0000001 ckBTC, which is burned, and a ckBTC minter fee when retrieving BTC. The latter is determined as follows:

* Under the conservative assumption that 1 BTC = 10,000 XDR, 1 billion cycles corresponds to 10 Satoshi (because 1 trillion cycles corresponds to 1 XDR).
* The [https://internetcomputer.org/docs/current/developer-docs/deploy/computation-and-storage-costs/ cost] to obtain a single EDCSA signature is approximately 21.54 billion cycles on a 28-node subnet, whereas sending a Bitcoin transaction costs 5 billion cycles plus 20 million cycles per byte.

Given these numbers, the cost to sign and send a transaction with <code>in</code> inputs and <code>out</code> outputs is

21.54b*in + 5b + tx_size*20m cycles
< 21.54b*in + 5b + (149*in + 35*out + 10)*20m cycles
< 24.52b*in +0.7b*out + 5.2b cycles
< 246*in + 7*out + 52 satoshi.

The formula <code>246*in + 7*out + 52</code> is used to determine the ckBTC minter’s fee in satoshi. Since every transaction has at least one input and one output, the fee is at least 305 Satoshi.

This conservative pricing strategy is used to subsidize the other endpoints, which are free of charge. Moreover, while the <code>retrieve_btc</code> endpoint is relatively expensive, the fee is typically still lower than the Bitcoin miner fee.

As mentioned above, there is also a KYT fee (currently 2000 Satoshi) when converting BTC to ckBTC and vice versa.

== ckBTC Minter API ==

The ckBTC minter provides the following endpoints:

* <code>get_btc_address</code>: Returns a specific Bitcoin address that the caller can use to obtain ckBTC by sending BTC to this address.
* <code>track_balance</code>: Instructs the ckBTC minter to track a certain Bitcoin address until funds are added, which will trigger the minting of ckBTC. '''NOTE: This endpoint is work-in-progress and not available yet.'''
* <code>update_balance</code>: Instructs the ckBTC minter to check the balance of a Bitcoin address and mint ckBTC into the account of the owner.
* <code>estimate_withdrawal_fee</code>: Returns a current estimate for the fee to be paid when retrieving a certain BTC amount.
* <code>get_deposit_fee</code>: Returns the fee charged when minting ckBTC.
* <code>get_withdrawal_account</code>: Returns a specific ckBTC account where the owner must transfer ckBTC before being able to retrieve BTC.
* <code>retrieve_btc</code>: Instructs the ckBTC minter to burn a certain ckBTC amount and send the corresponding BTC amount, minus fees, to a provided Bitcoin address.
* <code>retrieve_btc_status</code>: Returns the status of a previous <code>retrieve_btc</code> call.
* <code>get_minter_info</code>: Returns information about the ckBTC minter itself.
* <code>get_events</code>: Returns a set of events at the ckBTC minter.

The endpoints are discussed in more detail in the following sections.

==== <code>get_btc_address(owner: opt principal, subaccount: opt blob)</code> ====

The provided principal ID and subaccount are concatenated to form the derivation path for the [https://internetcomputer.org/docs/current/references/ic-interface-spec#ic-ecdsa_public_key ecdsa_public_key] function, which returns the derived public key. If no principal ID is provided, then the sender’s principal ID is used. If no subaccount is provided, then the default subaccount (all zeros) is used.

This public key is encoded as a pay-to-witness-public-key-hash (P2WPKH) Bitcoin address and returned as a text.

Note that the key derivation is not [https://en.bitcoin.it/wiki/BIP_0032 BIP-32] compliant where 31 bits are used for each derivation level. Instead, a single derivation is performed based on the full principal ID and subaccount. Since the derivation is deterministic, a canister can derive the Bitcoin address for a given principal ID and subaccount itself.

==== <code>track_balance(owner: opt principal, subaccount: opt blob)</code> ====
'''NOTE: This endpoint is work-in-progress and not available yet.'''

The ckBTC minter starts tracking the Bitcoin address derived from the provided principal ID and subaccount using the <code>get_btc_address</code> endpoint. If no principal ID is provided, then the sender’s principal ID is used. If no subaccount is provided, then the default subaccount (all zeros) is used.

The balance of the Bitcoin address is not tracked indefinitely. Tracking is stopped if either at least one new unspent transaction output (UTXO) is discovered or there is no new UTXO within a certain time interval (details about balance tracking are provided below).

If at least one new UTXO is discovered, the same amount of ckBTC tokens minus the KYT fee are minted and made available to the owner.

==== <code>update_balance(owner: opt principal, subaccount: opt blob)</code> ====

Instead of having the ckBTC minter track the balance of a Bitcoin address, the <code>update_balance</code> function can be invoked to instruct the ckBTC minter to check if there are new UTXOs for a particular Bitcoin address.

If there is at least one new UTXO, the corresponding ckBTC amount is minted, otherwise an error is returned.

The ckBTC minter effectively invokes this endpoint itself on a timer when the <code>track_balance</code> function is used.

==== <code>estimate_withdrawal_fee(amount: opt nat64)</code> ====
The endpoint returns an estimate for the fee that must be paid when retrieving the given BTC amount. Internally, a transaction is built (without valid signatures) to determine the fee, which consists of the Bitcoin miner fee, the ckBTC minter fee, and the KYT fee. If no amount is provided, it is assumed that three inputs are required to build the transaction.

If there is no change to the internal state of the ckBTC minter and the [https://github.com/dfinity/bitcoin-canister Bitcoin canister] before issuing the request to retrieve bitcoins, the fee will be exactly the returned estimate.

The fee can change when a) a new Bitcoin block is mined in the meantime, which causes the Bitcoin canister to update the Bitcoin miner fees or b) another retrieval request is handled first, spending some of the outputs that were used when estimating the fee.

==== <code>get_deposit_fee</code> ====
The endpoint returns the fee that the ckBTC minter charges for minting ckBTC when receiving new UTXOs. Currently, this fee is simply the KYT fee.

==== <code>get_withdrawal_account</code> ====

The function returns the caller’s withdrawal account, which is the account derived from the ckBTC minter’s principal ID and the subaccount derived from the caller’s principal ID.

A user can only retrieve BTC by first transferring ckBTC to this particular account.

==== <code>retrieve_btc(address: text, amount: nat64)</code> ====

The function instructs the ckBTC minter to burn the specified amount. Once the request is picked up from the queue, a Bitcoin transaction is created containing an output that transfers the requested amount (minus fees as discussed below) to the specified Bitcoin address.

Since Bitcoin retrieval is handled asynchronously, the function returns the block index of the transaction burning the ckBTC tokens.

==== <code>retrieve_btc_status(block_index: nat64)</code> ====

The status of a BTC retrieval request can be checked using this function. The different possible statuses are:

* <code>Unknown</code>: There is no information available on the ckBTC minter because there is no retrieval request associated with the given block index or the retrieval request is old and the corresponding information has already been removed.
* <code>Pending</code>: The BTC retrieval request is queued.
* <code>Signing</code>: The BTC retrieval request is acquiring all signatures to serve the BTC retrieval request.
* <code>Sending</code>: The Bitcoin transaction serving the BTC retrieval request is being sent.
* <code>Submitted</code>: The Bitcoin transaction has been transmitted. The transaction ID is returned.
* <code>AmountTooLow</code>: The BTC retrieval request could not be served because the Bitcoin miner fees are prohibitively high.
* <code>Confirmed</code>: The Bitcoin transaction serving the BTC retrieval request is confirmed inside the ckBTC minter, which happens when the transaction has at least the minimum required number of confirmations specified in the min_confirmations parameter above.

==== <code>get_minter_info</code> ====
The function returns the following parameters internal to the ckBTC minter:

* <code>kyt_fee</code>
* <code>min_confirmations</code>
* <code>retrieve_btc_min_amount</code>

==== <code>get_events(start: nat64, length: nat64)</code> ====

The ckBTC minter tracks the events that change its internal state. Given a start index and a length parameter, the ckBTC minter returns all events sequentially from the event at the given start index up to the event at index start+length-1.

Note that this endpoint is used for debugging purposes and there is no guarantee that the endpoint will continue to exist in this form.

== See also ==

* Code Bitcoin on the Internet Computer: https://internetcomputer.org/bitcoin-integration
* Chain-key tokens: https://internetcomputer.org/how-it-works/chain-key-tokens/
* Chain-key Bitcoin developer documents: https://internetcomputer.org/docs/current/developer-docs/integrations/bitcoin/ckbtc