Internet Computer Wiki - User contributions [en]

Exchange rate canister

2023-08-02T07:46:10Z

Yotam.harchol: Updated pricing

== Overview ==

The exchange rate canister (XRC) is a canister running on the [https://dashboard.internetcomputer.org/subnet/uzr34-akd3s-xrdag-3ql62-ocgoh-ld2ao-tamcv-54e7j-krwgb-2gm4z-oqe uzr34 system subnet] that provides exchange rates to requesting canisters.
A request comprises a base asset, a quote asset, and an optional (UNIX epoch) timestamp.
The base and quote asset can be any combination of cryptocurrency and fiat currency assets, for example, BTC/ICP, ICP/USD, or USD/EUR.
The timestamp parameter makes it possible to request historic rates. If no timestamp is provided in the request, the rate for the current time is returned.

The XRC constitutes an on-chain oracle for exchange rates, which is particularly useful for DeFi applications but can further add value to any application that requires exchange rate information.

The cycle minting canister of the [https://wiki.internetcomputer.org/wiki/NNS_Canisters NNS] will make use of the XRC to obtain up-to-date ICP/XDR rates, which it requires for the conversion of ICP to cycles.

== Usage ==

The canister ID of the XRC is <code>uf6dk-hyaaa-aaaaq-qaaaq-cai</code>.
A request of the form

type GetExchangeRateRequest = record {
base_asset: Asset;
quote_asset: Asset;
timestamp: opt nat64;
};

can be sent to the XRC, which replies with the following result:

type GetExchangeRateResult = variant {
Ok: ExchangeRate;
Err: ExchangeRateError;
};

An <code>Asset</code> is a record consisting of a symbol (for example, "ICP") and a class (either <code>Cryptocurrency</code> or <code>FiatCurrency</code>).
The full candid file can be found [https://github.com/dfinity/exchange-rate-canister/blob/main/src/xrc/xrc.did here].
The optional timestamp in the request must be a UNIX timestamp in seconds when provided. If no timestamp is provided, the timestamp corresponding to the start of the current minute is used.
Note that the granularity for requests is 1 minute, so seconds in a timestamp are ignored.

It is further worth nothing that some exchanges may not always have exchange rates available for the current minute. Depending on the use case, it may be advisable to use the start of the previous minute to increase the chance to get a response based on rates collected from all queried exchanges.

For every request, 1B cycles need to be sent along, otherwise an <code>ExchangeRateError::NotEnoughCycles</code> error is returned. The actual cost of the call depends on two factors, the requested asset types and the state of the internal exchange rate cache, as follows:

* If the request can be served from the cache, the actual cost is 20M cycles.
* If both assets are fiat currencies, the cost is 20M cycles as well.
* If one of the assets is a fiat currency or the cryptocurrency USDT, the cost is 260M cycles.
* If both assets are cryptocurrencies, the cost is 500M cycles.

The remaining cycles are returned to the requesting canister. Note that at least 1M cycles are charged even in case of an error in order to mitigate the risk of a denial-of-service attack.

== Technical Details ==

The following figure depicts the work flow when receiving a request.

[[File:XRC Flow Diagram.png|thumb|center|upright=3|Overview of the exchange rate canister work flow (bank icon from [https://www.flaticon.com/free-icon/bank_858170 flaticon.com]).]]

After receiving a request (step 1), the exchange rate for each cryptocurrency asset in the request with respect to the quote asset USDT is queried (for the timestamp in the request) from all supported exchanges using [[HTTPS outcalls]] if this rate is not already cached (step 2).
If a rate can be computed based on the query results received from the exchanges, it is inserted in the cache and returned to the requesting canister (step 3).
The ''median rate'' of all received rates is returned as it is not susceptible to outliers (unlike the average rate).

If a cryptocurrency/cryptocurrency base-quote pair B/Q was requested, the B/Q rate is derived from the queried B/USDT and Q/USDT rates.

The XRC queries daily foreign exchange (forex) rates from forex data providers automatically on a fixed schedule. Furthermore, the XRC queries multiple stablecoin rates automatically to derive the USD/USDT rate as follows. Given SC1/USDT, SC2/USDT, ... rates for a set of stablecoins SC1, SC2, ..., it uses the median of these rates as the USD/USDT rate. This rule is based on the assumption that at least half of the stablecoins in the set keep their peg to USD at any time, in which case the median rate is an adequate estimate for the USD/USDT rate.
Given the USD/USDT rate and the forex rates for fiat currencies other than USD, the requested rate can be computed for the case when one or more assets in the request are fiat currencies.

Since more requests to exchanges are required for cryptocurrency/cryptocurrency pairs, more cycles are charged for such requests.

As indicated in the figure above, the response to a successful request contains metadata in addition to the rate. The metadata contains the following fields:

* <code>decimals</code>: The rate is returned as a scaled 64-bit integer. The scaling factor is 10 to the power of <code>decimals</code>.
* <code>base_asset_num_received_rates</code>: The number of received rates for the base asset from all queried exchanges.
* <code>base_asset_num_queried_sources</code>: The number of queried exchanges for the base asset.
* <code>quote_asset_num_received_rates</code>: The number of received rates for the quote asset from all queried exchanges.
* <code>quote_asset_num_queried_sources</code>: The number of queried exchanges for the quote asset.
* <code>standard_deviation</code>: The standard deviation of all received rates for this request. Note that the standard deviation is scaled by the same factor as the rate itself.
* <code>forex_timestamp</code>: The timestamp of the beginning of the day for which the forex rates were retrieved, if any.

This additional information can be used to determine the trustworthiness of the received rate, for example by checking the number of rates that went into the computation of the rate and the standard deviation.
If the XRC receives largely inconsistent rates from exchanges, it returns an <code>ExchangeRateError::InconsistentRatesReceived</code> itself.

HTTPS outcalls

2023-07-26T13:13:29Z

Yotam.harchol: Added a note on outcalls being IPv6-only

'''On the Internet Computer blockchain, canister smart contracts can make HTTP outcalls to specified URLs, either to directly obtain off-chain data, or to interact with off-chain systems, such as Web 2.0 services or enterprise IT infrastructure. The results of these calls are processed and agreed by consensus, preventing nondeterminism. This avoids the need for trusted oracles and bridges.'''

Often, smart contract software needs to obtain real-world data, which originates from outside the secure and unstoppable on-chain universe that the blockchain that hosts them provides. Smart contracts may also need to interact with off-chain systems that are outside this universe. Because of the way blockchains work, historically this has presented major hurdles to blockchain developers.

For example, to obtain off-chain data, smart contracts have traditionally interacted with centrally-operated oracle services, such as [https://chain.link/ Chainlink]. These services are provided by trusted intermediaries, such as corporations, which perform the role of copying off-chain data onto the blockchain where it can be accessed by smart contracts. The problem is that these services must a) be trusted to be honest, and not get hacked, or otherwise become faulty, and b) be paid. Moreover, they cannot help when smart contracts need to ''interact'' with off-chain services, for example by calling into web-based APIs. To solve for these needs, the Internet Computer provides an "HTTPS outcall" feature.

HTTPS outcalls allow canister smart contracts hosted on the Internet Computer to request a URL, for example to download a time series recording the recent prices of an asset published by a centralized crypto exchange such as [https://pro.coinbase.com/ Coinbase Pro]. When this occurs, every node in the subnet blockchain hosting the smart contract requests the URL separately. Each node then locally passes the result they obtained to a special function implemented by the requesting canister smart contract using a [[query call]], which pre-processes the result with the aim of making it consistent with the results the other nodes have obtained and pre-processed (in our Coinbase example, since each node would request the time series at a slightly different moment, the results could be different).

If the pre-processed results obtained by query calls to the canister smart contract are sufficiently consistent across all the nodes, the result is agreed by consensus, and provided back to the smart contract that requested the URL so that it can continue trustlessly processing the original smart contract call (TX).

In order to trigger an action in an off-chain system, a smart contract may include a cryptographic [[chain key]] signature in its request for a URL. This allows the target service to validate that the request it has received was generated by a genuine smart contract execution that was agreed by consensus. In such architectures, when an off-chain service receives a valid request for a URL, it must take care to only execute it once, since many nodes will make the same request, and for each subsequent request after the first, it should return exactly the same result.

Note: Currently, HTTPS outcalls are only possible to IPv6 hosts. That is, hosts with a valid AAAA DNS record. HTTPS outcalls to IPv4-only hosts will fail on mainnet. The reason for this is that IC nodes only have an IPv6 address assigned, and the IC network is IPv6-only.
== Architecture ==
The canister HTTPS outcalls feature has been implemented as an extension of the IC protocol stack, particularly its consensus layer. The possibility of the IC protocol stack allowing for such extensions shows the powerful architecture of the IC and its consensus protocol. This section gives details on the architecture of the feature: which components of the stack are needed to be extended and which new components were required and explain a protocol flow through the stack.

=== HTTPS Outcall Request Lifecycle ===
[[File:HTTPS outcall request lifecycle1.png|thumb|700px|HTTPS outcall request lifecycle]]
The figure above shows the process an HTTPS outcall request goes through, along with the new components that were added to make this feature work. The numbered arrows indicate the steps as follows:
# A canister makes an HTTPS outcall request to the management canister in the execution layer. The request is stored in the replicated state of the corresponding subnet.
# A new component in the Consensus layer, called the “HTTP pool manager”, is reading state changes and keeps track of outstanding HTTPS requests.
#Whenever the HTTP pool manager sees a new request, it forwards it to the networking layer, to a new component that is called “HTTP adapter shim”. This is a relatively lightweight component that is responsible for communicating with the “HTTP adapter”, which is a separate process running alongside the replica process, but is isolated for security reasons.
# The HTTP adapter shim forwards the request to the HTTP adapter using RPC.
# The HTTP adapter on each node issues the requested HTTPS request to the remote server.
# A response is returned from the server to each HTTP adapter.
# Each HTTP adapter returns the response to the HTTP adapter shim component.
# The HTTP adapter shim invokes an optional transform function on the calling canister. The purpose of this function is explained in detail below, but in short, it should help in making all responses exactly the same so that the subnet can reach consensus on them.
# The HTTP adapter shim forwards the transformed response to the HTTP pool manager
# The Consensus layer then distributes shares of the response to all peers, so that the block maker can see that enough peers received exactly the same response as it has received.
# The block maker then includes the response in a block.
# The response is made available to the execution layer.
# A callback is invoked to return the response to the canister asynchronously.

== How to reach consensus on result? ==

As explained above, every node makes a given HTTP request to the target server and receives a response. There are multiple reasons why the responses are not necessarily the same on all replicas for the same API query:
* Typical implementations of web servers add variable elements to otherwise equal content, e.g., timestamps or unique identifiers. In this case, the actual content, e.g., requested asset prices, can be the same in each response, while those variable fields differ.
* Not all APIs are implemented such that they are nicely queryable to return the same response data in each response. E.g., financial data APIs may return data elements in different order in different responses or may have different starting timestamps of responses.
In order for the replicas to agree on a single response value as part of consensus, the different responses need to be equal. To achieve this property, each replica invokes a so-called transformation function on its received response and continues processing with the transformed response. The transformed response is used for the attempt to achieve consensus as explained earlier by first broadcasting an artifact corresponding to the transformed response through which a block making replica can observe whether a given response has a sufficient number of equal transformed responses.

Next, there are some examples on how consensus can be reached for HTTP responses. Different cases are illustrated below with an example of a simple weather API.

=== All responses are equal ===

The simplest case is that of all responses received by replicas being equal. In this case, no transformation function is needed as the responses themselves are already equal and consensus can be achieved on them.

[[File:all_responses_equal.png|center|700px|Identical responses – easy to reach consensus]]

=== Some responses are not equal ===

Assuming the case that less than one third of the responses can be arbitrarily different to the others and the others are all equal. The different responses can be a result of the server responding differently or nodes being compromised and falsifying the received (correct) response. This case is handled exactly as above by not requiring a transformation function and the IC consensus protocol taking care of the subset of deviating responses and going with the majority of replicas.

[[File:some_responses_not_equal.png|center|700px|At least two thirds of responses are equal – easy to reach consensus]]

=== Variable parts in all responses ===

The most general case is that of each response returned by the HTTP server being different due to the reasons outlined further above, e.g., variable parts being contained therein. This case requires a transformation function to "normalize" the responses to be pairwise equal (for at least two thirds of the replicas). The transformed responses then can be consented on by the protocol as in the cases above. Clearly, less than one third of the responses can still be arbitrarily different as in the case above and consensus can still be reached.

[[File:variable_parts_in_all_responses.png|center|700px|Identical responses after transformation – easy to reach consensus]]

As can be seen, all the above cases can be handled easily with an extension of the IC's consensus protocol. The important part is that the canister developer needs to provide the transformation function if required, i.e., in the typical case of using APIs. Refer to the feature documentation for a recipe to write a transformation function and further information to help developers getting started with the feature.

==See Also==
* '''The Internet Computer project website (hosted on the IC): [https://internetcomputer.org/ internetcomputer.org]'''

IC P2P (peer to peer) layer

2022-11-10T09:36:03Z

Yotam.harchol: Created page with "The peer-to-peer layer of the IC is responsible for delivering messages between peers within subnets. It is designed to ensure efficient delivery of messages even in the prese..."

The peer-to-peer layer of the IC is responsible for delivering messages between peers within subnets. It is designed to ensure efficient delivery of messages even in the presence of malicious activity.

== Artifacts ==
Each client of the P2P layer, i.e., components in the layers above it like Consensus, maintains a collection of artifacts (artifact pool). Artifacts are messages that are exchanged between peers within a subnet to create, validate and agree on blocks. These can be, for example, consensus block proposals, ingress messages from users, or signature shares of responses for the HTTPS outcalls feature. Nodes distribute these artifacts to their peers so all peers have the same view of the state of the subnet. This allows nodes to include ingress messages sent to other peers when they create block proposals and to catch up efficiently if they have been disconnected or joining a new subnet.

The P2P layer is invoked by the artifact pools when an artifact needs to be delivered to peers. This can happen when the node itself generates a new artifact (e.g., a new block proposal), or when it wants to ''relay'' an artifact it received from another peer (for example, we would want to relay block proposals even when all peers are connected to all peers, since a malicious peer can send one block proposal to a subset of peers, and another (or none) to another subset. This can slow down a subnet. Relaying proposals prevents this). Relay also helps in cases when some nodes are only connected to some peers. This can happen when a node’s ISP changes its BGP peering settings.

== Adverts ==
[[File:P2p.jpg|thumb|700px|Caption: Instead of sending an artifact from multiple peers, peers send adverts (smaller messages) and the recipient requests the artifact from a small subset of its peers.]]
In order to save bandwidth, which would be wasted by sending an artifact to a peer that already has the artifact in its pool, the P2P layer does not send out artifacts immediately to all peers when requested by a client to broadcast an artifact. Instead, it creates a message called ''advert'', which is a very small message with the hash of the artifact and some more metadata, and broadcasts this advert to all peers. Other peers then see the advert and decide whether they would like to download the corresponding artifact from that peer, and if so, they send an explicit ''request'' message to that peer, which will respond with the artifact itself. This approach clearly trades throughput for latency, but for the requirements of the IC’s consensus protocol this is desired.

To share the bandwidth fairly among all peers and to avoid running out of memory, the number of in-flight requests is limited.

== Chunks ==
Some artifacts, like state sync artifacts, are too big to be sent as a single chunk. For these artifacts, a single advert represents a set of ''chunks'' that together make up a complete artifact. When a downloaded artifact is ''chunkable'', the P2P layer will attempt to download the corresponding chunks from multiple peers which advertised the corresponding artifact. This speeds up the download and better utilizes the bandwidth.

== Verification of Artifacts ==
Each advert contains an ''integrity hash'' of the corresponding artifact. The integrity hash is the result of applying a cryptographic hash function over the content of the artifact. When an artifact download is complete, the same function is applied locally on the downloaded content, and only if the result matches the advertised hash value, the artifact is processed and provided to the corresponding artifact pool of the corresponding client.

It is important to note that this is only a first layer of defense: a malicious node can send an advert with some integrity hash, and then an artifact that matches the advertised hash. This will pass all checks in the P2P layer, and the client will then have to notice that the artifact does not meet its requirements, e.g., is not signed correctly. The clients in the IC implementation always perform validation checks to catch such attacks before processing artifacts further or relaying them to other peers.

Chunkable artifacts are validated both per-chunk and as a whole, where the corresponding client is responsible for per-chunk validation.

== Retransmission Requests ==
In some cases, for example when a node notices an error in receiving adverts, or when it joins a new subnet, it sends out a ''retransmission request'' to its peers. The request can go to one specific peer (e.g., when a connectivity issue is detected with that peer), or to all peers (e.g., when the node joins). The request contains information about the node’s current state, and peers can respond with adverts that can help the node make progress from its current state, as reported in the retransmission request.

While retransmission requests are meant for helping nodes to catch up on missed adverts, they may also issue them periodically to make sure they haven’t missed anything.

== Transport ==
The underlying Transport component of the P2P layer is responsible for maintaining the actual connections between peers in a subnet. It creates TLS over TCP connections between the peers, where both sides authenticate using their private keys.

Each peer has its public key stored in the registry canister of the Network Nervous System (NNS). The registry canister also contains the most up-to-date subnet membership information (which nodes belong to which subnet), as well as historic information. Each node learns its own membership, as well as who its peers are, what their IP addresses are, and what their public keys are, by querying the registry canister. Nodes always make sure they only connect to peers in their subnet by enforcing two-way authentication when establishing the TLS connections.

Subnet memberships change over time, with nodes being added and removed from subnets regularly based on NNS accepted proposals. The Transport component continuously tracks these changes and adjusts the connections accordingly. In some cases, nodes need to maintain connections to peers that are no longer in the subnet record of the newest registry canister entries, while consensus still considers them to be part of the subnet, and thus the Transport component allows for such connections until they are no longer required.

The Transport component also provides mechanisms for keeping the connection alive, quickly identifies connection issues (using a heartbeat mechanism), and automatically reconnects when the connection breaks. Whenever a TCP connection is idle for more than 200ms, a heartbeat message is sent over. On the receive side, when no data (including heartbeats) is received for more than 5s, the connection is being dropped and reconnected. This is done to avoid waiting for a possibly very long timeout duration of the TCP protocol, which sometimes happens upon Internet routing change events. After a connection is re-established, a retransmission request is sent to the corresponding peer.

File:P2p.jpg

2022-11-10T09:31:21Z

Yotam.harchol:

p2p

File:HTTPS outcall request lifecycle1.png

2022-10-05T06:03:09Z

Yotam.harchol: Yotam.harchol uploaded a new version of File:HTTPS outcall request lifecycle1.png

HTTPS outcall request lifecycle

HTTPS outcalls

2022-10-05T04:43:34Z

Yotam.harchol:

'''On the Internet Computer blockchain, canister smart contracts can make HTTP outcalls to specified URLs, either to directly obtain off-chain data, or to interact with off-chain systems, such as Web 2.0 services or enterprise IT infrastructure. The results of these calls are processed and agreed by consensus, preventing nondeterminism. This avoids the need for trusted oracles and bridges.'''

Often, smart contract software needs to obtain real-world data, which originates from outside the secure and unstoppable on-chain universe that the blockchain that hosts them provides. Smart contracts may also need to interact with off-chain systems that are outside this universe. Because of the way blockchains work, historically this has presented major hurdles to blockchain developers.

For example, to obtain off-chain data, smart contracts have traditionally interacted with centrally-operated oracle services, such as [https://chain.link/ Chainlink]. These services are provided by trusted intermediaries, such as corporations, which perform the role of copying off-chain data onto the blockchain where it can be accessed by smart contracts. The problem is that these services must a) be trusted to be honest, and not get hacked, or otherwise become faulty, and b) be paid. Moreover, they cannot help when smart contracts need to ''interact'' with off-chain services, for example by calling into web-based APIs. To solve for these needs, the Internet Computer provides an "HTTPS outcall" feature.

HTTPS outcalls allow canister smart contracts hosted on the Internet Computer to request a URL, for example to download a time series recording the recent prices of an asset published by a centralized crypto exchange such as [https://pro.coinbase.com/ Coinbase Pro]. When this occurs, every node in the subnet blockchain hosting the smart contract requests the URL separately. Each node then locally passes the result they obtained to a special function implemented by the requesting canister smart contract using a [[query call]], which pre-processes the result with the aim of making it consistent with the results the other nodes have obtained and pre-processed (in our Coinbase example, since each node would request the time series at a slightly different moment, the results could be different).

If the pre-processed results obtained by query calls to the canister smart contract are sufficiently consistent across all the nodes, the result is agreed by consensus, and provided back to the smart contract that requested the URL so that it can continue trustlessly processing the original smart contract call (TX).

In order to trigger an action in an off-chain system, a smart contract may include a cryptographic [[chain key]] signature in its request for a URL. This allows the target service to validate that the request it has received was generated by a genuine smart contract execution that was agreed by consensus. In such architectures, when an off-chain service receives a valid request for a URL, it must take care to only execute it once, since many nodes will make the same request, and for each subsequent request after the first, it should return exactly the same result.

== Architecture ==
The canister HTTPS outcalls feature has been implemented as an extension of the IC protocol stack, particularly its consensus layer. The possibility of the IC protocol stack allowing for such extensions shows the powerful architecture of the IC and its consensus protocol. In this section we give details on the architecture of the feature: We show which components of the stack needed to be extended and which new components were required and explain a protocol flow through the stack.

=== HTTPS Outcall Request Lifecycle ===
[[File:HTTPS outcall request lifecycle1.png|thumb|700px|HTTPS outcall request lifecycle]]
The figure above shows the process an HTTPS outcall request goes through, along with the new components that were added to make this feature work. The numbered arrows indicate the steps as follows:
# A canister makes an HTTPS outcall request to the management canister in the execution layer. The request is stored in the replicated state of the corresponding subnet.
# A new component in the Consensus layer, called the “HTTP pool manager”, is reading state changes and keeps track of outstanding HTTPS requests.
#Whenever the HTTP pool manager sees a new request, it forwards it to the networking layer, to a new component that is called “HTTP adapter shim”. This is a relatively lightweight component that is responsible for communicating with the “HTTP adapter”, which is a separate process running alongside the replica process, but is isolated for security reasons.
# The HTTP adapter shim forwards the request to the HTTP adapter using RPC.
# The HTTP adapter on each node issues the requested HTTPS request to the remote server.
# A response is returned from the server to each HTTP adapter.
# Each HTTP adapter returns the response to the HTTP adapter shim component.
# The HTTP adapter shim invokes an optional transform function on the calling canister. The purpose of this function is explained in detail below, but in short, it should help in making all responses exactly the same so that the subnet can reach consensus on them.
# The HTTP adapter shim forwards the transformed response to the HTTP pool manager
# The Consensus layer then distributes shares of the response to all peers, so that the block maker can see that enough peers received exactly the same response as it has received.
# The block maker then includes the response in a block.
# The response is made available to the execution layer.
# A callback is invoked to return the response to the canister asynchronously.

== How to reach consensus on result? ==

As explained above, every node makes a given HTTP request to the target server and receives a response. There are multiple reasons why the responses are not necessarily the same on all replicas for the same API query:
* Typical implementations of web servers add variable elements to otherwise equal content, e.g., timestamps or unique identifiers. In this case, the actual content, e.g., requested asset prices, can be the same in each response, while those variable fields differ.
* Not all APIs are implemented such that they are nicely queryable to return the same response data in each response. E.g., financial data APIs may return data elements in different order in different responses or may have different starting timestamps of responses.
In order for the replicas to agree on a single response value as part of consensus, the different responses need to be equal. To achieve this property, each replica invokes a so-called transformation function on its received response and continues processing with the transformed response. The transformed response is used for the attempt to achieve consensus as explained earlier by first broadcasting an artifact corresponding to the transformed response through which a block making replica can observe whether a given response has a sufficient number of equal transformed responses.

We next look at some examples on how consensus can be reached for HTTP responses. We illustrate the different cases below with an example of a simple weather API.

=== All responses are equal ===

The simplest case is that of all responses received by replicas being equal. In this case, no transformation function is needed as the responses themselves are already equal and consensus can be achieved on them.

[[File:all_responses_equal.png|center|700px|Identical responses – easy to reach consensus]]

=== Some responses are not equal ===

Assuming the case that less than one third of the responses can be arbitrarily different to the others and the others are all equal. The different responses can be a result of the server responding differently or nodes being compromised and falsifying the received (correct) response. This case is handled exactly as above by not requiring a transformation function and the IC consensus protocol taking care of the subset of deviating responses and going with the majority of replicas.

[[File:some_responses_not_equal.png|center|700px|At least two thirds of responses are equal – easy to reach consensus]]

=== Variable parts in all responses ===

The most general case is that of each response returned by the HTTP server being different due to the reasons outlined further above, e.g., variable parts being contained therein. This case requires a transformation function to "normalize" the responses to be pairwise equal (for at least two thirds of the replicas). The transformed responses then can be consented on by the protocol as in the cases above. Clearly, less than one third of the responses can still be arbitrarily different as in the case above and we can still reach consensus.

[[File:variable_parts_in_all_responses.png|center|700px|Identical responses after transformation – easy to reach consensus]]

As can be seen, all the above cases can be handled easily with an extension of the IC's consensus protocol. The important part is that the canister developer needs to provide the transformation function if required, i.e., in the typical case of using APIs. We refer the reader to the feature documentation for a recipe to write a transformation function and further information to help developers getting started with the feature.

File:HTTPS outcall request lifecycle1.png

2022-10-05T04:43:13Z

Yotam.harchol:

HTTPS outcall request lifecycle

HTTPS outcalls

2022-09-28T12:34:57Z

Yotam.harchol:

'''On the Internet Computer blockchain, canister smart contracts can make HTTP outcalls to specified URLs, either to directly obtain off-chain data, or to interact with off-chain systems, such as Web 2.0 services or enterprise IT infrastructure. The results of these calls are processed and agreed by consensus, preventing nondeterminism. This avoids the need for trusted oracles and bridges.'''

Often, smart contract software needs to obtain real-world data, which originates from outside the secure and unstoppable on-chain universe that the blockchain that hosts them provides. Smart contracts may also need to interact with off-chain systems that are outside this universe. Because of the way blockchains work, historically this has presented major hurdles to blockchain developers.

For example, to obtain off-chain data, smart contracts have traditionally interacted with centrally-operated oracle services, such as [https://chain.link/ Chainlink]. These services are provided by trusted intermediaries, such as corporations, which perform the role of copying off-chain data onto the blockchain where it can be accessed by smart contracts. The problem is that these services must a) be trusted to be honest, and not get hacked, or otherwise become faulty, and b) be paid. Moreover, they cannot help when smart contracts need to ''interact'' with off-chain services, for example by calling into web-based APIs. To solve for these needs, the Internet Computer provides an "HTTPS outcall" feature.

HTTPS outcalls allow canister smart contracts hosted on the Internet Computer to request a URL, for example to download a time series recording the recent prices of an asset published by a centralized crypto exchange such as [https://pro.coinbase.com/ Coinbase Pro]. When this occurs, every node in the subnet blockchain hosting the smart contract requests the URL separately. Each node then locally passes the result they obtained to a special function implemented by the requesting canister smart contract using a [[query call]], which pre-processes the result with the aim of making it consistent with the results the other nodes have obtained and pre-processed (in our Coinbase example, since each node would request the time series at a slightly different moment, the results could be different).

If the pre-processed results obtained by query calls to the canister smart contract are sufficiently consistent across all the nodes, the result is agreed by consensus, and provided back to the smart contract that requested the URL so that it can continue trustlessly processing the original smart contract call (TX).

In order to trigger an action in an off-chain system, a smart contract may include a cryptographic [[chain key]] signature in its request for a URL. This allows the target service to validate that the request it has received was generated by a genuine smart contract execution that was agreed by consensus. In such architectures, when an off-chain service receives a valid request for a URL, it must take care to only execute it once, since many nodes will make the same request, and for each subsequent request after the first, it should return exactly the same result.

== Architecture ==
The canister HTTPS outcalls feature has been implemented as an extension of the IC protocol stack, particularly its consensus layer. The possibility of the IC protocol stack allowing for such extensions shows the powerful architecture of the IC and its consensus protocol. In this section we give details on the architecture of the feature: We show which components of the stack needed to be extended and which new components were required and explain a protocol flow through the stack.

=== HTTPS Outcall Request Lifecycle ===
[[File:HTTPS outcall request lifecycle.png|thumb|700px|HTTPS outcall request lifecycle]]
The figure above shows the process an HTTPS outcall request goes through, along with the new components that were added to make this feature work. The numbered arrows indicate the steps as follows:
# A canister makes an HTTPS outcall request to the management canister in the execution layer. The request is stored in the replicated state of the corresponding subnet.
# A new component in the Consensus layer, called the “HTTP pool manager”, is reading state changes and keeps track of outstanding HTTPS requests.
#Whenever the HTTP pool manager sees a new request, it forwards it to the networking layer, to a new component that is called “HTTP adapter shim”. This is a relatively lightweight component that is responsible for communicating with the “HTTP adapter”, which is a separate process running alongside the replica process, but is isolated for security reasons.
# The HTTP adapter shim forwards the request to the HTTP adapter using RPC.
# The HTTP adapter on each node issues the requested HTTPS request to the remote server.
# A response is returned from the server to each HTTP adapter.
# Each HTTP adapter returns the response to the HTTP adapter shim component.
# The HTTP adapter shim invokes an optional transform function on the calling canister. The purpose of this function is explained in detail below, but in short, it should help in making all responses exactly the same so that the subnet can reach consensus on them.
# The HTTP adapter shim forwards the transformed response to the HTTP pool manager
# The Consensus layer then distributes shares of the response to all peers, so that the block maker can see that enough peers received exactly the same response as it has received.
# The block maker then includes the response in a block.
# The response is made available to the execution layer.
# A callback is invoked to return the response to the canister asynchronously.

== How to reach consensus? ==

As explained above, every node makes a given HTTP request to the target server and receives a response. There are multiple reasons why the responses are not necessarily the same on all replicas for the same API query:
* Typical implementations of web servers add variable elements to otherwise equal content, e.g., timestamps or unique identifiers. In this case, the actual content, e.g., requested asset prices, can be the same in each response, while those variable fields differ.
* Not all APIs are implemented such that they are nicely queryable to return the same response data in each response. E.g., financial data APIs may return data elements in different order in different responses or may have different starting timestamps of responses.
In order for the replicas to agree on a single response value as part of consensus, the different responses need to be equal. To achieve this property, each replica invokes a so-called transformation function on its received response and continues processing with the transformed response. The transformed response is used for the attempt to achieve consensus as explained earlier by first broadcasting an artifact corresponding to the transformed response through which a block making replica can observe whether a given response has a sufficient number of equal transformed responses.

We next look at some examples on how consensus can be reached for HTTP responses. We illustrate the different cases below with an example of a simple weather API.

=== All responses are equal ===

The simplest case is that of all responses received by replicas being equal. In this case, no transformation function is needed as the responses themselves are already equal and consensus can be achieved on them.

[[File:all_responses_equal.png|center|700px|Identical responses – easy to reach consensus]]

=== Some responses are not equal ===

Assuming the case that less than one third of the responses can be arbitrarily different to the others and the others are all equal. The different responses can be a result of the server responding differently or nodes being compromised and falsifying the received (correct) response. This case is handled exactly as above by not requiring a transformation function and the IC consensus protocol taking care of the subset of deviating responses and going with the majority of replicas.

[[File:some_responses_not_equal.png|center|700px|At least two thirds of responses are equal – easy to reach consensus]]

=== Variable parts in all responses ===

The most general case is that of each response returned by the HTTP server being different due to the reasons outlined further above, e.g., variable parts being contained therein. This case requires a transformation function to "normalize" the responses to be pairwise equal (for at least two thirds of the replicas). The transformed responses then can be consented on by the protocol as in the cases above. Clearly, less than one third of the responses can still be arbitrarily different as in the case above and we can still reach consensus.

[[File:variable_parts_in_all_responses.png|center|700px|Identical responses after transformation – easy to reach consensus]]

As can be seen, all the above cases can be handled easily with an extension of the IC's consensus protocol. The important part is that the canister developer needs to provide the transformation function if required, i.e., in the typical case of using APIs. We refer the reader to the feature documentation for a recipe to write a transformation function and further information to help developers getting started with the feature.

File:HTTPS outcall request lifecycle.png

2022-09-28T12:30:28Z

Yotam.harchol:

HTTPS outcall request lifecycle