Difference between revisions of "HTTPS outcalls"
Diego.prats (talk | contribs) |
|||
| Line 13: | Line 13: | ||
== Architecture == | == Architecture == | ||
| − | The canister HTTPS outcalls feature has been implemented as an extension of the IC protocol stack, particularly its consensus layer. The possibility of the IC protocol stack allowing for such extensions shows the powerful architecture of the IC and its consensus protocol. | + | The canister HTTPS outcalls feature has been implemented as an extension of the IC protocol stack, particularly its consensus layer. The possibility of the IC protocol stack allowing for such extensions shows the powerful architecture of the IC and its consensus protocol. This section gives details on the architecture of the feature: which components of the stack are needed to be extended and which new components were required and explain a protocol flow through the stack. |
=== HTTPS Outcall Request Lifecycle === | === HTTPS Outcall Request Lifecycle === | ||
| Line 39: | Line 39: | ||
In order for the replicas to agree on a single response value as part of consensus, the different responses need to be equal. To achieve this property, each replica invokes a so-called transformation function on its received response and continues processing with the transformed response. The transformed response is used for the attempt to achieve consensus as explained earlier by first broadcasting an artifact corresponding to the transformed response through which a block making replica can observe whether a given response has a sufficient number of equal transformed responses. | In order for the replicas to agree on a single response value as part of consensus, the different responses need to be equal. To achieve this property, each replica invokes a so-called transformation function on its received response and continues processing with the transformed response. The transformed response is used for the attempt to achieve consensus as explained earlier by first broadcasting an artifact corresponding to the transformed response through which a block making replica can observe whether a given response has a sufficient number of equal transformed responses. | ||
| − | + | Next, there are some examples on how consensus can be reached for HTTP responses. Different cases are illustrated below with an example of a simple weather API. | |
| Line 58: | Line 58: | ||
=== Variable parts in all responses === | === Variable parts in all responses === | ||
| − | The most general case is that of each response returned by the HTTP server being different due to the reasons outlined further above, e.g., variable parts being contained therein. This case requires a transformation function to "normalize" the responses to be pairwise equal (for at least two thirds of the replicas). The transformed responses then can be consented on by the protocol as in the cases above. Clearly, less than one third of the responses can still be arbitrarily different as in the case above and | + | The most general case is that of each response returned by the HTTP server being different due to the reasons outlined further above, e.g., variable parts being contained therein. This case requires a transformation function to "normalize" the responses to be pairwise equal (for at least two thirds of the replicas). The transformed responses then can be consented on by the protocol as in the cases above. Clearly, less than one third of the responses can still be arbitrarily different as in the case above and consensus can still be reached. |
[[File:variable_parts_in_all_responses.png|center|700px|Identical responses after transformation – easy to reach consensus]] | [[File:variable_parts_in_all_responses.png|center|700px|Identical responses after transformation – easy to reach consensus]] | ||
| − | As can be seen, all the above cases can be handled easily with an extension of the IC's consensus protocol. The important part is that the canister developer needs to provide the transformation function if required, i.e., in the typical case of using APIs. | + | As can be seen, all the above cases can be handled easily with an extension of the IC's consensus protocol. The important part is that the canister developer needs to provide the transformation function if required, i.e., in the typical case of using APIs. Refer to the feature documentation for a recipe to write a transformation function and further information to help developers getting started with the feature. |
==See Also== | ==See Also== | ||
* '''The Internet Computer project website (hosted on the IC): [https://internetcomputer.org/ internetcomputer.org]''' | * '''The Internet Computer project website (hosted on the IC): [https://internetcomputer.org/ internetcomputer.org]''' | ||
Revision as of 10:46, 27 February 2023
On the Internet Computer blockchain, canister smart contracts can make HTTP outcalls to specified URLs, either to directly obtain off-chain data, or to interact with off-chain systems, such as Web 2.0 services or enterprise IT infrastructure. The results of these calls are processed and agreed by consensus, preventing nondeterminism. This avoids the need for trusted oracles and bridges.
Often, smart contract software needs to obtain real-world data, which originates from outside the secure and unstoppable on-chain universe that the blockchain that hosts them provides. Smart contracts may also need to interact with off-chain systems that are outside this universe. Because of the way blockchains work, historically this has presented major hurdles to blockchain developers.
For example, to obtain off-chain data, smart contracts have traditionally interacted with centrally-operated oracle services, such as Chainlink. These services are provided by trusted intermediaries, such as corporations, which perform the role of copying off-chain data onto the blockchain where it can be accessed by smart contracts. The problem is that these services must a) be trusted to be honest, and not get hacked, or otherwise become faulty, and b) be paid. Moreover, they cannot help when smart contracts need to interact with off-chain services, for example by calling into web-based APIs. To solve for these needs, the Internet Computer provides an "HTTPS outcall" feature.
HTTPS outcalls allow canister smart contracts hosted on the Internet Computer to request a URL, for example to download a time series recording the recent prices of an asset published by a centralized crypto exchange such as Coinbase Pro. When this occurs, every node in the subnet blockchain hosting the smart contract requests the URL separately. Each node then locally passes the result they obtained to a special function implemented by the requesting canister smart contract using a query call, which pre-processes the result with the aim of making it consistent with the results the other nodes have obtained and pre-processed (in our Coinbase example, since each node would request the time series at a slightly different moment, the results could be different).
If the pre-processed results obtained by query calls to the canister smart contract are sufficiently consistent across all the nodes, the result is agreed by consensus, and provided back to the smart contract that requested the URL so that it can continue trustlessly processing the original smart contract call (TX).
In order to trigger an action in an off-chain system, a smart contract may include a cryptographic chain key signature in its request for a URL. This allows the target service to validate that the request it has received was generated by a genuine smart contract execution that was agreed by consensus. In such architectures, when an off-chain service receives a valid request for a URL, it must take care to only execute it once, since many nodes will make the same request, and for each subsequent request after the first, it should return exactly the same result.
Architecture
The canister HTTPS outcalls feature has been implemented as an extension of the IC protocol stack, particularly its consensus layer. The possibility of the IC protocol stack allowing for such extensions shows the powerful architecture of the IC and its consensus protocol. This section gives details on the architecture of the feature: which components of the stack are needed to be extended and which new components were required and explain a protocol flow through the stack.
HTTPS Outcall Request Lifecycle
The figure above shows the process an HTTPS outcall request goes through, along with the new components that were added to make this feature work. The numbered arrows indicate the steps as follows:
- A canister makes an HTTPS outcall request to the management canister in the execution layer. The request is stored in the replicated state of the corresponding subnet.
- A new component in the Consensus layer, called the “HTTP pool manager”, is reading state changes and keeps track of outstanding HTTPS requests.
- Whenever the HTTP pool manager sees a new request, it forwards it to the networking layer, to a new component that is called “HTTP adapter shim”. This is a relatively lightweight component that is responsible for communicating with the “HTTP adapter”, which is a separate process running alongside the replica process, but is isolated for security reasons.
- The HTTP adapter shim forwards the request to the HTTP adapter using RPC.
- The HTTP adapter on each node issues the requested HTTPS request to the remote server.
- A response is returned from the server to each HTTP adapter.
- Each HTTP adapter returns the response to the HTTP adapter shim component.
- The HTTP adapter shim invokes an optional transform function on the calling canister. The purpose of this function is explained in detail below, but in short, it should help in making all responses exactly the same so that the subnet can reach consensus on them.
- The HTTP adapter shim forwards the transformed response to the HTTP pool manager
- The Consensus layer then distributes shares of the response to all peers, so that the block maker can see that enough peers received exactly the same response as it has received.
- The block maker then includes the response in a block.
- The response is made available to the execution layer.
- A callback is invoked to return the response to the canister asynchronously.
How to reach consensus on result?
As explained above, every node makes a given HTTP request to the target server and receives a response. There are multiple reasons why the responses are not necessarily the same on all replicas for the same API query:
- Typical implementations of web servers add variable elements to otherwise equal content, e.g., timestamps or unique identifiers. In this case, the actual content, e.g., requested asset prices, can be the same in each response, while those variable fields differ.
- Not all APIs are implemented such that they are nicely queryable to return the same response data in each response. E.g., financial data APIs may return data elements in different order in different responses or may have different starting timestamps of responses.
In order for the replicas to agree on a single response value as part of consensus, the different responses need to be equal. To achieve this property, each replica invokes a so-called transformation function on its received response and continues processing with the transformed response. The transformed response is used for the attempt to achieve consensus as explained earlier by first broadcasting an artifact corresponding to the transformed response through which a block making replica can observe whether a given response has a sufficient number of equal transformed responses.
Next, there are some examples on how consensus can be reached for HTTP responses. Different cases are illustrated below with an example of a simple weather API.
All responses are equal
The simplest case is that of all responses received by replicas being equal. In this case, no transformation function is needed as the responses themselves are already equal and consensus can be achieved on them.
Some responses are not equal
Assuming the case that less than one third of the responses can be arbitrarily different to the others and the others are all equal. The different responses can be a result of the server responding differently or nodes being compromised and falsifying the received (correct) response. This case is handled exactly as above by not requiring a transformation function and the IC consensus protocol taking care of the subset of deviating responses and going with the majority of replicas.
Variable parts in all responses
The most general case is that of each response returned by the HTTP server being different due to the reasons outlined further above, e.g., variable parts being contained therein. This case requires a transformation function to "normalize" the responses to be pairwise equal (for at least two thirds of the replicas). The transformed responses then can be consented on by the protocol as in the cases above. Clearly, less than one third of the responses can still be arbitrarily different as in the case above and consensus can still be reached.
As can be seen, all the above cases can be handled easily with an extension of the IC's consensus protocol. The important part is that the canister developer needs to provide the transformation function if required, i.e., in the typical case of using APIs. Refer to the feature documentation for a recipe to write a transformation function and further information to help developers getting started with the feature.
See Also
- The Internet Computer project website (hosted on the IC): internetcomputer.org