Authored by Colin Schwarz

Light Client Task Force Call #1

Notes and Transcript

On November 4th, 2019, ChainSafe and the Lodestar team sponsored the first ever Light Client Task Force call. The purpose of the call was to bring together individuals who are interested in Eth2.0 light clients and start a discussion to drive further collaboration and research in this area. This first call was led by Lodestar tech lead Cayman Nava and featured a talk and Q&A by Zsolt Felfoldi who is a leader in the Eth1 light client field. A full recording of the call can be found here.

Below is a summary of the call and a transcript. An attendance list is included at the bottom of this article.

Eth1 Light Clients

We have seen the Eth1 light client system struggle to reach production. However, the light client tech that has reached production has proven very beneficial and has really increased the usability of Ethereum. A good example is Beam Sync, which was recently implemented in Trinity and decreased sync times by about 100x. Consequently, getting things like this into production for Eth2.0 will be incredibly important. This call is just the beginning of the discussion

This call features a Q&A with Zsolt Felfoldi who has been working on Eth1 light clients for several years and has created a light client server incentivization model. This is important work and may be a good foundation for light clients on Eth2.

Zsolt's Light Client Incentivization Model

Overview

An obvious problem with light clients is that they require a server to serve requests. This server is usually a full node so there need to be incentives for people to run the servers that light clients rely on. In some cases, it is possible for clients to just cache information. This is mainly header information but information needed for syncing will also be useful here. This is an asymmetrical setup: in some cases some kind of compensation is needed. However, it would be ideal to create a market where some level of free service is also available because it is really hard to join a network with a light client if you need to find a full client to pay for data etc. every time. Free service is important but Zsolt wanted a model where you could get more bandwidth and more reliable connections by paying someone.

Further Challenges and Considerations

• Another big challenge is doing this in a decentralized way, without having to rely on any sort of centralized registry to keep track of reliable servers and the like. Despite this, Zsolt still wanted light clients to be able to find good servers.

• If you want to pay someone in a decentralized setup, you already need to built up some sort of trust because the server is just a network address and a public key and if you make a payment to a fake server, your money could disappear.

• This has led to a model where servers usually give out a certain amount of their capacity for free, to build trust. Then, once they have too many clients, the server offers these clients the option to buy a better service with better connection and bandwidth.

• One challenge here was measuring service quality and pricing based on this. It is hard to set a fixed price table of requests that accurately represents the actual difficulty of serving requests. This is mainly because of caching. Even if it's the same type of request sometimes it's 100x cheaper at one time than at another.

• If you want to know how much service you received and if the price for that service actually made sense, you would need long-term statistics and comparisons to the performance of other servers which is all somewhat subjective. It wouldn't make sense to try to set an objective and globally valid score system for service quality for each server because if we assume that each server is able to answer a given request then service quality mostly comes down to consistently good response times. However, a response time depends not only on the server but also on the network.

• It would be ideal for clients to find servers that are both good and close, to reduce network latency.

Payment Channels

• Any light client model that requires payments for better service as discussed above will require payment channels.

• At this point it is safe to say that there are a variety of options for this on Ethereum, from simple on-chain payments, to more complex layer two solutions like Raiden and Plasma.

• The underlying issue is thus not the payment channels themselves, but how to determine how much each request is worth given the problems discussed above regarding how noisy the ecosystem is.

• Zsolt: "It is hard to determine the value of each individual request because both the difficulty and response times of each request are really noisy and fluctuate widely. We need longer term averages to make these values meaningful and to make really meaningful market decisions."

Szolt's Solution

It is good to do some pre-negotiation of rate limiting policies. Szolt created this LES server that features a rate limiting model. Right now it's called "client side flow control." LES was created a few years ago but with recent updates to the server it's really starting to do a reasonably good job. The model ensures that if needed it can straddle each client so it doesn't get overloaded. The thing the LES team wanted to avoid initially, and something that happened frequently, was that clients started connecting and sending more and more requests. Then the server had too many requests queued up which caused response times to increase significantly to several seconds and resulted in a lot of time-outs. The LES server has since been modified to deal with these issues. The LES server can assign a scalar capacity value to each client which defines a parameter for a rate limiting policy so the server can dynamically adjust its total capacity allowance in order to prevent frequent overloads. It can also guarantee reasonably low response times in the majority of cases. In order to do this, the server does a negotiation. The first part of the negotiation is getting into the client server. The pricing can be based both on connection time and individual requests.

To make a market, their has to be competition for scarce resources. The primary scarce resource for each server is total capacity allowance. Trying to compete with pricing out other clients on a short timescale is not a good model. The team wanted to plan into the short and near future and use some kind of service token which makes things more predictable and incentivizes honesty and good market behavior. The connection priority can be based on the amount of service tokens held by each client which is a commitment to use the service in the near future. The service is also now interested in hosting the clients who enjoyed its services. The use of service tokens helps to smooth out the demand over time for server capacity. This token helps to decentralize reputation building in the ecosystem: clients prefer servers who have been offering good servers and servers prefer those clients who found them good service in the past.

Q&A on Zsolt's Incentivization Model

(transcript edited for clarity and conciseness, items not in quotes are summaries)

Eric "Suppose I was a new user trying to use Ethereum. Wouldn't this be kind of a chicken and egg thing? I know you mentioned there's some services that would be free but let's say I'm trying to purchase my first ETH so I can use the network, but I wouldn't have ETH in the first place to pay light client servers?"

Zsolt "Yes, that's a very important case. My model tries to make the availability of free service an inherent property of the whole network because we really need some amount of free service because of being able to on-board new clients."

• Another problem is the trust problem. New clients will not want to pay for servers they do not know cause these servers could run away with the money. To solve this, if a client finds a server that is new to them then they can try and connect for free at first, even if the server usually charges. Free service is limited but if you keep trying you can eventually connect. A client only has to pay once they have received a sufficient amount of free service. This is how servers start to build trust. "Every server that wants to charge money must provide free service until it has enough clients competing for the service."

• If a client received free service, they will temporarily receive a negative token balance to their IP address so that newcomers who want free service will be prioritized above those who have already received it.

• It is guaranteed that for a certain percentage of time, the necessary percentage of funds required to connect to a server will be zero or negative which means that it will be possible for at least a few clients to connect for free.

• "It is in the interest of the server to control their prices in such a way as to ensure this because otherwise they will not get new clients and also because if they have too many paying clients then these paying clients will not find this service honest because they will not be able to spend their tokens if there is too much competition. This is the theory, we will see how it works in practice"

Mikhail "I have a question about servers providing bad or incorrect data and still getting paid for it. Is it assumed that servers will be penalized for bad actions?"

Zsolt

In a peer to peer setup the promises are not really directly enforceable. The way to penalize servers is that they will have to provide free service in order to get an amount of clients who want to pay them and this is what they are going to lose if they are providing a worthless service. So the implicit promise of a connection is that the server will respond with a consistent response time. How long that response time is, is not entirely the server's responsibility so it's also up to the client to determine whether that response time is good enough for it or not. Perhaps this is something that needs a little bit more formal analysis but the goal is that by starting to cheat, giving out too many tokens, not caring about service quality or giving bad responses, the server should not be able to gain more money by starting to breach its promises and abuse its reputation.

Johns "How do we handle reorgs in that setup? What if a light client decides to swap between the free services offered by multiple servers?"

Zsolt

This model where we assign limited capacity to each client, this has an advantage that if you want to pay and if you want to get a higher bandwidth priority connection then you can also buy a low bandwidth from multiple servers and get the same bandwidth or even more than a single one. It's really good if the client is connected to as many servers as possible. This has the advantage that the client will receive the new head announcements from all the servers. If it gets a different head announcement from all the servers then it will download the new block headers from both of them. The client can check the consensus rules and see which is the longest chain and it will request data from the currently longest valid chain. So it is always the light clients responsibility to follow the most valuable chain. Right now in the proof of work case our security assumption is that maybe attackers can create one fake block header but they certainly cannot create a chain of fake blocks with valid proof of work that stays the longest chain for any meaningful amount of time. So once the client finds the longest chain that's what it's interested in, so the other servers who don't find the longest chain quick enough don't have a chance to provide service so they will be practically useless for the client.

Cayman "I've got a question about the payment side. So you were mentioning payment channels, you mentioned selling these short term tokens. It seems like each server has its own tokens, potentially each server could have its own payment channel. It seems like it maybe makes sense to enshrine or standardize this payment aspect, otherwise you kind of have a really big headache on the client side of how to actually pay. I'm assuming you don't want this intrusive system where you have to continually be confirming and continually hitting that buy button. You kind of want to load up and then it runs for a few months. Can we talk about how you see the payment side working out?"

Zsolt

Sure. I see two different questions here. One of them is the payment technology. So actually, until now I preferred to not enshrine one specific payment technology into the protocol, especially because the best potential ways to pay are still a little bit experimental and here I mean Raiden and also Swarm is working on a multistep payment technology. So the issue here is that it's really easy to use one to one payment channels but they already have some trust issues in terms of double spending or over spending so the simple case is a one to one channel but then setting up a new channel... the issue is that setting up a new channel already costs an on-chain transaction which might be acceptable for now. Actually, since my model is not based on paying for every single request but paying for longer amounts and also converting Ether to tokens is something that can happen gradually, so if I can already trust the server with say 0.1 Eth then I can just always buy the amount of service tokens I really need. The issue is the ideal payment technology would be something that can do root payments through multiple machines and use already existing payment channels to find the new payment directions. An advantage of Raiden is that if you find a bunch of good servers and you buy a little from all of them, you won't have a high initial cost of paying for a new direction. These technologies are also in progress and I don't want to tie the system to just one of them. The way I'm doing it right now is that the servers will announce which sort of currencies they are willing to accept. Maybe Eth will be the preferred payment currency, maybe DAI, I don't know. Initially servers could accept simple on-chain payment but later, payment channel technologies can compete for being the primarily used technology for light client incentivization.... I want to leave it up to the market.

The other thing I want to talk about here is how the user will see this and use this (the payment methods). This is something that can be automatic. The user can set an average response time vs cost per request or cost per request or cost per connection second preference curve. I don't know how it will be most convenient but the user can specify some preferences and the client strategy can automatically choose a server to pay for and decide that OK, now I'm going to pay for this server. It depends on the payment technology but the signing of transactions will be done by some sort of external system. Geth already has an external signer called Clef but Raiden also has its own stuff. So the user just needs to set an allowance that for example, in the next month I want to only spend two dollars, and then they leave the rest up to the client so it should be automatic.

Cayman "How is the cost of a request calculated for the server?"

Zsolt

The request prices are determined by the server, and the client will decide whether it's good or not. The way it currently works, and what I found workable right now, is that the server announces a maximum cost estimate for each request type, and also depending on the amount of data requested. The actual individual request prices are individually determined by the server and that's because of caching. So it depends on the nature of the request. I think if we want to make this market efficient, the best way is to let the server decide how expensive a request was in retrospect within a maximum limit. Since all the performance vs price ratios can only be determined over at least several minutes, the client will decide "ok, now I've paid this much for this request, I'm sending a similar request to another server and now I can compare how much this one has charged compared to the other." The way the server actually decides the prices is related to the way it controls its maximum capacity because there's usually an important bottleneck in the system, or a few important bottlenecks. So right now with the Go Ethereum light server, the basis of pricing is the time spent serving the request in a single thread, which is probably not ideal but is good enough for now and maybe we can refine this in the future. Usually it's limited by database access so maybe we can also find some tricks for database access but the point is that this server tries to determine the request prices nominated in service tokens in a way that is proportional to the actual usage of the most important bottlenecks and it is trying to ensure that it can process requests with a similar total cost per second to make sure it is fully utilized. So the server tries to keep the purchasing power of its service stable by pricing requests in such a way that the requests processable per second have a more or less constant token price per second. This makes the expandability of tokens predictable so then the server will know how many tokens it can sell for the near future. So basically the pricing is up to the server and deciding if it's a good price is up to the client.

Cayman "Is the protocol for broadcasting the current amount of tokens and current capacity and those kind of things worked out specifically or is that something still left to do?"

Zsolt

So that's already part of the LES protocol even though it's only used for free service. Even with the free service as I mentioned, we already have this negative balance thing so we already use this pricing method even though we don't do actual payments. So this rate limiting policy is implemented in a certain way in LES and I'm not saying that it should be exactly the same for future protocols but we can evaluate how well it will perform when we introduce actual payments. The actual payment and price negotiations is the thing that's currently being implemented. Right now I'm adding a quick UDP request response thing to the servers that can quickly tell any client about the current token prices and the necessary amount of priority to connect at the moment at a certain capacity. This is something where I want to implement a minimum complexity thing that LES needs right now that can work with payments while thinking about making it more general if necessary. So yes, that part is figured out already but nothing is set in stone forever.

Note: Some of this was documented a long time ago. You can ask Zsolt for links on Telegram. The details of the price negotiations are going to be documented when Zsolt releases this code. Szolt is interested in feedback and suggestions on this

Alex "I had a question on numbers on LES servers on the network today ?"

Szolt "At the moment I have no idea. We have a really ugly technical difficulty with the DHT that light clients and servers are using currently. One of my colleagues on the Go team, Felix Langer is working on improving the DHT right now. We want to know more about the current state of the network but this DHT is not really reliable and at this moment I'm not sure if you can find all the servers easily. This discovery protocol DHT is something that many people are interested in right now. It has this codename called Discv5, so we have already completely redesigned it and soon we can give you a number of currently existing light servers but right now this is a technical difficulty that clients cannot find all of the servers on the network."

Mikhail "What part of Discv5 is used to discover the servers?"

Zsolt "What do you mean what part?"

Mikhail "I mean do you use topic discovery or ENR capabilities?"

Zsolt "We are trying to use the topic discovery. The other codename of Discv5 was topic discovery so this was the intended feature but right now only LES is using it so the topic capability is not really relevant. Well, it is a little bit relevant because light servers do exist on multiple chains so right now the advertised topics are used to identify the correct chain and the correct kind of service. What currently exists and what is currently used by light clients and light servers is a previous experimental version of Discv5 and it doesn't have ENRs integrated yet. The final version of Discv5 will also have ENS integrated so that also tells about server quality and the chain the server is operating on."

Cayman "Any final thoughts?"

Zsolt "I would just wrap it up by saying I am really glad that we are converging around light clients and that people are also interested in incentivization and I'd be really happy if I got a lot of feedback and if it's critical feedback because I don't want to stick to any of my ideas too adamantly and I'd like to move this whole ecosystem forward and I think making this peer to peer network layer and all the infrastructure we have making this more marketized, that's really key to making it more scalable. Making a good consensus algorithm and making sharding viable, that's also important but I think this thing just cannot work without making the underlying infrastructure more marketized. I just wanted to share all my experiences with light client development and the reasoning around incentivization. Thank you for coming and listening and asking questions!"

Cayman's Closing Remarks

"We will catch you next month. We will put together an agenda and find another topic to focus our conversation and make the most of a synchronized call. One thought that I have immediately is potentially covering Vitalik's recent Magician's post about being able to store the latest roughly four months of state in Eth1 as a potential topic but I'm interested to hear everyone else's ideas. So hit me up on Telegram or Discord and we will catch you next month."

Johns "Perhaps we can also discuss the Javascript implementation that you guys have with Lodestar. I would like to get a bit more of an understanding about how that operates as a light client. Like to bridge Lodestar and these other topics your doing."

Cayman "Ya, I'd be happy to talk about that. A lot of it's in a research phase, a prototyping phase. We're building on the Eth2 effort and a lot of the Eth2 light client consensus level data is only available after phase 1, and so as that is hardening we're prototyping. There's a little bit of back and forth there, we don't really have anything to play around with..."

Johns "Not really for playing around, what I'm trying to wrap my head around is another light client because I know Lodestar specifically, from what I understand, it's a full Javascript implementation of the Beacon chain and the shards and validators and stuff but in my mind I can't quite bridge an understanding of how that's applicable to light clients."

Cayman "Ok, so you're talking about like ‘what is the structure of an Eth2 light client, how could you go about that"?

Johns "Ya."

Cayman "Ok, that is a great idea for another topic."

Alex "If we wanted to do something similar for today we could try to get Felix involved and talk about Discv5 and how it pertains to light clients."

Cayman "Excellent idea, seems like it's a very important part of the puzzle."

Thanks for reading. Our next light client call will be held tomorrow, December 4th, at 10am Eastern! Here is a link to the Github issue.

Attendance

• Aidan - Interested in light clients and interoperability

• Alex - Works on trinity, excited by beam sync stuff

• Danny - Eth researcher, light clients will be even more important with sharding

• Ed - Interested in Eth2 client work, wanted to stay up to date

• Johns - Working on distributed storage system, interested in Lodestar as a light client

• Eric - Lodestar dev, excited for light clients to increase usability

• Marin - Lodestar dev, wants to help integrate light client libp2p, syncing light clients

• Matt - Works on Quilt team, light clients important to stateless execution

• Mikhail - With Harmony, interested in light clients for a long time

• Proto - EF research team, involved with tooling and SSZ merkle proofs

• Terence - Prysmatic labs, wants to learn more about Eth1 light clients

• Vitalik - Light client part of Eth2 spec, shard redesign has light client part built in

• Zahari - Nimbus team, light clients are crucial in their goal to bring Ethereum to phones

• Greg - Lodestar dev, ChainSafe

• Zaki - Tendermint, core primitive of IBC protocol is a light client, looking to expand IBC

• Zsolt - Go Ethereum developer, started go Ethereum light client, will help us learn from Eth1 experience with light client development. Incentivization is key problem with light clients.