Difference between revisions of "HTTPS outcalls"
m |
m |
||
| Line 10: | Line 10: | ||
In order to trigger an action in an off-chain system, a smart contract may include a cryptographic [[chain key]] signature in its request for a URL. This allows the target service to validate that the request it has received was generated by a genuine smart contract execution that was agreed by consensus. In such architectures, when an off-chain service receives a valid request for a URL, it must take care to only execute it once, since many nodes will make the same request, and for each subsequent request after the first, it should return exactly the same result. | In order to trigger an action in an off-chain system, a smart contract may include a cryptographic [[chain key]] signature in its request for a URL. This allows the target service to validate that the request it has received was generated by a genuine smart contract execution that was agreed by consensus. In such architectures, when an off-chain service receives a valid request for a URL, it must take care to only execute it once, since many nodes will make the same request, and for each subsequent request after the first, it should return exactly the same result. | ||
| + | |||
| + | |||
| + | == Architecture == | ||
| Line 26: | Line 29: | ||
The simplest case is that of all responses received by replicas being equal. In this case, no transformation function is needed as the responses themselves are already equal and consensus can be achieved on them. | The simplest case is that of all responses received by replicas being equal. In this case, no transformation function is needed as the responses themselves are already equal and consensus can be achieved on them. | ||
| − | [[File:all_responses_equal.png|center| | + | [[File:all_responses_equal.png|center|700px|Identical responses – easy to reach consensus]] |
| Line 33: | Line 36: | ||
Assuming the case that less than one third of the responses can be arbitrarily different to the others and the others are all equal. The different responses can be a result of the server responding differently or nodes being compromised and falsifying the received (correct) response. This case is handled exactly as above by not requiring a transformation function and the IC consensus protocol taking care of the subset of deviating responses and going with the majority of replicas. | Assuming the case that less than one third of the responses can be arbitrarily different to the others and the others are all equal. The different responses can be a result of the server responding differently or nodes being compromised and falsifying the received (correct) response. This case is handled exactly as above by not requiring a transformation function and the IC consensus protocol taking care of the subset of deviating responses and going with the majority of replicas. | ||
| − | [[File:some_responses_not_equal.png|center| | + | [[File:some_responses_not_equal.png|center|700px|At least two thirds of responses are equal – easy to reach consensus]] |
| Line 40: | Line 43: | ||
The most general case is that of each response returned by the HTTP server being different due to the reasons outlined further above, e.g., variable parts being contained therein. This case requires a transformation function to "normalize" the responses to be pairwise equal (for at least two thirds of the replicas). The transformed responses then can be consented on by the protocol as in the cases above. Clearly, less than one third of the responses can still be arbitrarily different as in the case above and we can still reach consensus. | The most general case is that of each response returned by the HTTP server being different due to the reasons outlined further above, e.g., variable parts being contained therein. This case requires a transformation function to "normalize" the responses to be pairwise equal (for at least two thirds of the replicas). The transformed responses then can be consented on by the protocol as in the cases above. Clearly, less than one third of the responses can still be arbitrarily different as in the case above and we can still reach consensus. | ||
| − | [[File:variable_parts_in_all_responses.png|center| | + | [[File:variable_parts_in_all_responses.png|center|700px|Identical responses after transformation – easy to reach consensus]] |
As can be seen, all the above cases can be handled easily with an extension of the IC's consensus protocol. The important part is that the canister developer needs to provide the transformation function if required, i.e., in the typical case of using APIs. We refer the reader to the feature documentation for a recipe to write a transformation function and further information to help developers getting started with the feature. | As can be seen, all the above cases can be handled easily with an extension of the IC's consensus protocol. The important part is that the canister developer needs to provide the transformation function if required, i.e., in the typical case of using APIs. We refer the reader to the feature documentation for a recipe to write a transformation function and further information to help developers getting started with the feature. | ||
Revision as of 09:48, 28 September 2022
On the Internet Computer blockchain, canister smart contracts can make HTTP outcalls to specified URLs, either to directly obtain off-chain data, or to interact with off-chain systems, such as Web 2.0 services or enterprise IT infrastructure. The results of these calls are processed and agreed by consensus, preventing nondeterminism. This avoids the need for trusted oracles and bridges.
Often, smart contract software needs to obtain real-world data, which originates from outside the secure and unstoppable on-chain universe that the blockchain that hosts them provides. Smart contracts may also need to interact with off-chain systems that are outside this universe. Because of the way blockchains work, historically this has presented major hurdles to blockchain developers.
For example, to obtain off-chain data, smart contracts have traditionally interacted with centrally-operated oracle services, such as Chainlink. These services are provided by trusted intermediaries, such as corporations, which perform the role of copying off-chain data onto the blockchain where it can be accessed by smart contracts. The problem is that these services must a) be trusted to be honest, and not get hacked, or otherwise become faulty, and b) be paid. Moreover, they cannot help when smart contracts need to interact with off-chain services, for example by calling into web-based APIs. To solve for these needs, the Internet Computer provides an "HTTPS outcall" feature.
HTTPS outcalls allow canister smart contracts hosted on the Internet Computer to request a URL, for example to download a time series recording the recent prices of an asset published by a centralized crypto exchange such as Coinbase Pro. When this occurs, every node in the subnet blockchain hosting the smart contract requests the URL separately. Each node then locally passes the result they obtained to a special function implemented by the requesting canister smart contract using a query call, which pre-processes the result with the aim of making it consistent with the results the other nodes have obtained and pre-processed (in our Coinbase example, since each node would request the time series at a slightly different moment, the results could be different).
If the pre-processed results obtained by query calls to the canister smart contract are sufficiently consistent across all the nodes, the result is agreed by consensus, and provided back to the smart contract that requested the URL so that it can continue trustlessly processing the original smart contract call (TX).
In order to trigger an action in an off-chain system, a smart contract may include a cryptographic chain key signature in its request for a URL. This allows the target service to validate that the request it has received was generated by a genuine smart contract execution that was agreed by consensus. In such architectures, when an off-chain service receives a valid request for a URL, it must take care to only execute it once, since many nodes will make the same request, and for each subsequent request after the first, it should return exactly the same result.
Architecture
How to reach consensus?
As explained above, every node makes a given HTTP request to the target server and receives a response. There are multiple reasons why the responses are not necessarily the same on all replicas for the same API query:
- Typical implementations of web servers add variable elements to otherwise equal content, e.g., timestamps or unique identifiers. In this case, the actual content, e.g., requested asset prices, can be the same in each response, while those variable fields differ.
- Not all APIs are implemented such that they are nicely queryable to return the same response data in each response. E.g., financial data APIs may return data elements in different order in different responses or may have different starting timestamps of responses.
In order for the replicas to agree on a single response value as part of consensus, the different responses need to be equal. To achieve this property, each replica invokes a so-called transformation function on its received response and continues processing with the transformed response. The transformed response is used for the attempt to achieve consensus as explained earlier by first broadcasting an artifact corresponding to the transformed response through which a block making replica can observe whether a given response has a sufficient number of equal transformed responses.
We next look at some examples on how consensus can be reached for HTTP responses. We illustrate the different cases below with an example of a simple weather API.
All responses are equal
The simplest case is that of all responses received by replicas being equal. In this case, no transformation function is needed as the responses themselves are already equal and consensus can be achieved on them.
Some responses are not equal
Assuming the case that less than one third of the responses can be arbitrarily different to the others and the others are all equal. The different responses can be a result of the server responding differently or nodes being compromised and falsifying the received (correct) response. This case is handled exactly as above by not requiring a transformation function and the IC consensus protocol taking care of the subset of deviating responses and going with the majority of replicas.
Variable parts in all responses
The most general case is that of each response returned by the HTTP server being different due to the reasons outlined further above, e.g., variable parts being contained therein. This case requires a transformation function to "normalize" the responses to be pairwise equal (for at least two thirds of the replicas). The transformed responses then can be consented on by the protocol as in the cases above. Clearly, less than one third of the responses can still be arbitrarily different as in the case above and we can still reach consensus.
As can be seen, all the above cases can be handled easily with an extension of the IC's consensus protocol. The important part is that the canister developer needs to provide the transformation function if required, i.e., in the typical case of using APIs. We refer the reader to the feature documentation for a recipe to write a transformation function and further information to help developers getting started with the feature.