A group of friends is out for beers. They are all working in the same field and, as usual, somebody is telling a story from work — how they did this or that and that thing failed spectacularly and that other guy was a complete jerk and so on. You overhear them as you wait for your date to come back from the restroom and, naturally, one of two things happens: either you understand what they are talking about and silently laugh along with them, or you get bored quickly by the technical jargon and go back to checking your Facebook feed. This is such a story. If you are also a software engineer (or even more so a blockchains developer) then you might enjoy reading about the sorrows of trying to set up my very own Ethereum local testnet.

But first, a little context. This year, I have the fortune to be the Teaching Assistant at the Blockchains & Distributed Ledgers course of the University here. The course has a semester project, which is supposed to familiarize the students with Ethereum development, that is to say they have to interact with deployed contracts and code at least one contract themselves.

One thing to keep in mind is the need for a sanitized environment. Normally, I would just deploy a bunch of contracts to Ropsten and use the faucets to get some testnet Ether. However, working in such open environment, where anybody can come around and play with your contracts, makes a few palms sweatier than what the beautiful Scottish weather usually allows for.

The solution? Deploy your own Ethereum clone and let the kids go nuts in there. Simple, right?

“I have seen how the foundations of the world are laid”

First things first, we need to set up an Ethereum node. There are a lot of forum posts, Stack Exchange comments, blogs etc that describe this process — a special shout out to this extremely helpful article is much needed. However, for reasons that will become apparent by the end of this article, I felt necessary to add my own little contribution to this stream of literature.

Back to our work now, with a bit of googling we find that the best two candidates are Geth and Parity — let’s go with the former. I run Ubuntu, but I don’t really like to install packages when not absolutely needed. Following the installation instructions from the wiki, I opt for building from source. So I install Go, clone the Geth repo and make geth. Works like a charm.

(Important note: At the time this post is published, the latest release of Geth is 1.18.17, so all mentions of the repo refer to this commit.)

Let’s deploy our very own Ethereum now. I take the Geth binary from /build/bin to a separate folder and, following the instructions on the repo, I copy the following genesis.json file that they so kindly provide.

{
  "config": {
        "chainId": 0,
        "homesteadBlock": 0,
        "eip155Block": 0,
        "eip158Block": 0
    },
  "alloc"      : {},
  "coinbase"   : "0x0000000000000000000000000000000000000000",
  "difficulty" : "0x20000",
  "extraData"  : "",
  "gasLimit"   : "0x2fefd8",
  "nonce"      : "0x0000000000000042",
  "mixhash"    : "0x0000000000000000000000000000000000000000000000000000000000000000",
  "parentHash" : "0x0000000000000000000000000000000000000000000000000000000000000000",
  "timestamp"  : "0x00"
}

The guide only mentions what to do with the nonce and alloc parameters, without a link to a description of what the other parameters are. Weird. Let’s check the online tutorials. After a bit of search I find that the chainId is actually the id of our private network, so I really don’t want to set it to some value that might be already used and accidentally connect to an existing network. I use 83293829389243 — should be unique enough. I also set alloc to assign a million Ether to the address I have already created in Metamask. Finally, I set the difficulty to 1, since I don’t really want to start with heavy proof-of-work, and run ./geth --datadir "./datadir" init init.json. Seems to work.

Next step is to boot the node and start mining. The Geth repo gives some instructions about bootnodes, which rather confuse me than help, so, following the tutorials, I simply run ./geth --rpc --rpccorsdomain "*" --datadir "./datadir" --nodiscover --networkid 83293829389243 --etherbase "<my_address>" console and actually get a console — amazing!

(Important note: Make sure your chainId and your networkid are the exact same. Also make sure that they are not too big. If you put too big a value, either of the commands might complain or things will simply break and you will not be sure why — when in doubt, try a smaller id.)

I should also note now that there are A LOT of hidden stuff behind this command. For example, the RPC endpoint by default runs on localhost:8545, an IPC endpoint also starts by default on localhost:30303, the RCP API default interfaces are db, eth, net, web3 (for information on each one check here) and many others. I should also mention a very useful flag, --rpcaddr "<your public IP", which allows you to connect remotely via RPC to the node (assuming you have a static IP and have opened the RPC port of course).

Now that I have a console, I can start the miner with miner.start() — all good, blocks flooding in.

Let’s deal with the difficulty issue now. As we know from theory, as more blocks start piling the difficulty threshold will readjust in order to tend towards 1 block/17 seconds. This means that eventually both our CPU will run at 100% and block production will have that rate. I don’t want to stress my CPU for no reason though. In order to avoid the difficulty readjustment, I change a line of code in Geth: in the file /consensus/ethash/consensus.go I simply set the function CalcDifficulty in line 313 to return big.NewInt(1); and delete all other statements in it.

The final challenge now is to adjust the miner’s execution. When the miner runs, because of the minimum difficulty, it will produce blocks at an enormous rate and fill the chain (and the disk) with empty blocks. A slight problem when trying to change this behaviour is that, in order to start/stop the miner, you need to have a Geth console, which is not very handy for automated scripts like the one I want to write. The good thing is that you can preload a JavaScript script by simply adding the flag --preload "mine.js" when running the Geth command above.mine.js is the following custom script:

function sleep () {
    miner.stop();
    setTimeout(work, 10000);
}

function work () {
    if (txpool.status.pending !== 0) miner.start();
    setTimeout(sleep, 2000);
}

work();

This script runs in a constant loop, where the miner checks if there are any pending transactions that need to be mined and, if so, starts mining (I don’t really care about queued transactions, but it is easy to handle them also). After 2 seconds, it stops, sleeps for 10 seconds and then wakes up and starts from the beginning. You can change the intervals to what you need, but I have found that, on a mediocre cloud server, 2 seconds are on average enough to mine a couple of blocks.

At this point we have our node up and running in a screen on our server. So, I connect to it using Metamask and send a transaction to see everything running smoothly — I even see on the Geth console a few mined blocks.

And now…

A small rant about Metamask: the interface of Metamask is good-ish in general. An infuriating bug (or feature?) appears if you have added your own RPC private network and connect to one of the pre-defined networks (like the mainnet). Then, your custom network disappears from the list and you have to add it again — EVERY.FUCKING.TIME. Also, in many cases, Metamask fails with super generic error messages, like “Internal JSON RPC error” (for example, one such case is if you have Metamask open in two different browsers/devices and one does not listen the transactions of the other; in this case, the nonce that the first uses in its transactions is old, so the transactions will remained queued). Just remember, whenever you are not sure what the problem is, simply remove Metamask and re-install it.

A typical debugging process for Metamask

The final thing to do now is test my chain by deploying a contract. The quickest way to write a smart contract is by using Remix, as it is the easiest-to-use beginner’s tool (Truffle etc are much more sophisticated) and, combined with Metamask, offers a first development experience which is as good as one can hope to have. That does not mean that it is good though.

Let’s assume that I belong to the 60% of people who use Google Chrome (I really don’t, since I prefer Firefox, but bare with me here). I open Remix and after a few seconds am greeted with the following message.

The first thing a Chrome user sees of Remix

That’s as a bad start as it can get. To make things worse, let’s also assume that I am too loyal to Google and still don’t want to use Firefox. So, I simply press OK and start writing my first contract. I press compile — and it compiles! I deploy it — and it deploys successfully, transaction receipt and all! Let me repeat that: I ignore the error message, the compiler is in a non-sane state and Remix still allows me to compile like nothing happened and runs smoothly after. I really don’t want to know what kind of black magic runs under the hood, but Remix, honestly, get your shit together.

In any case, it seems that we actually have our node up and running now and can deploy contracts. I give the students the instructions on how to connect to my node and let them go nuts.

“… and the golden towers of Byzantium would be my tomb.”

A couple of days later, a rather innocent question comes from a student: “If the require statement fails, is all the gas that I defined in the transaction used or is the remaining returned to me?”. Confident and full of hubris, I immediately respond “Search for the differences between require and assert in Solidity”. However, something clicks and I am curious about why he would ask this, so I test it myself.

I deploy a test contract with a require, send a transaction which I know will throw— and all gas that I defined is consumed!

A few hours pass by and I find myself (almost literally) banging my head against the wall. In retrospect, a mistake that I made during this debugging process is that I tested everything on my local network. In the meantime, the discussions at the office revolve around the compiler version, the Solidity version, the Geth version etc. It is pretty indicative of the whole mentality of a software ecosystem when the developers have to worry about what tool’s stable version they are using.

After some time, I test the contract at Remix’s Javascript VM and Ropsten. At that point two things happen. First, I understand that the problem is with my local chain, since it works as I expect in these environments. Second, I notice a comment on one of the tens of tabs that I have opened in my browser. The comment includes just one word: “Byzantium”. And I get the epiphany.

About a year ago, Ethereum hard forked to the “Byzantium” release. Among other things, this release changed the gas rules. Prior to it, require and assert were practically the same, whereas after it, require eats only the gas that is used up to the point that the statement fails, and not all of the gas defined in the transaction. So, I think to myself, is it possible that my node is running pre-Byzantium consensus?

First, I check Geth. My local repo is checked out at the latest release, which is a few days old — I doubt that the latest Geth does not support Byzantium. Second, I check my genesis file. It is exactly as the Geth repository says, and also as the various tutorials on how to set up your local testnet say. However, these tutorials are mostly a year old. Byzantium was not active then, so what if they don’t care for it? And how do I enable Byzantium by default in my chain?

Bingo. There actually does exist a parameter, called ByzantiumBlock, which can be added to the config part of the genesis json file and specifies from which block onwards Byzantium will start to apply. Of course, the official docs don’t feel the need to specify this — why the fuck would they, right?

At this point I am too exhausted to try too hard to sync a new Byzantium node with my old one, like the link above describes. I try it a couple of times, but, after an initial exchange of messages and headers, it just refuses to accept the old chain. If anybody knows how to do this, please write it in the comments. My solution was to simply scrap the old chain and set a new one, with Byzantium enabled from block 0using the genesis file below. Fortunately, I have not noticed any further problems with it — yet.

{
  "config": {
        "chainId": 83293829389243,
        "homesteadBlock": 0,
        "ByzantiumBlock": 2,
        "eip155Block": 0,
        "eip158Block": 0
    },
  "alloc"      : {
    "<my address>":   {"balance": "100000000000000000000000"}
  },
  "coinbase"   : "0x0000000000000000000000000000000000000000",
  "difficulty" : "1",
  "extraData"  : "",
  "gasLimit"   : "0x2fefd8",
  "nonce"      : "0x0000000000000042",
  "mixhash"    : "0x0000000000000000000000000000000000000000000000000000000000000000",
  "parentHash" : "0x0000000000000000000000000000000000000000000000000000000000000000",
  "timestamp"  : "0x00"
}

Conclusion

The key takeaway from my short time in the Ethereum development space is that the powers that be simply don’t care about backwards compatibility or documentation. If you want to play with these tools, be it from the lowest level, like deploying the platform, to the highest level, like interacting with contracts, prepare for bad documentation, obscure parameters, guides that will be valid for only a couple of months and, especially, lots and lots of cursing. If you are reading this a few months from now, if you are new to the space, I sincerely hope that what I have shown here is still applicable. I also sincerely believe that it won’t be — you have my sympathies.