Demo of Three-Node Two-Channel Setup in Hyperledger FabricKC TamBlockedUnblockFollowFollowingApr 6OverviewSince the demo of a multi-channel setup (link) on a localhost is published last December, I keep receiving questions about a multi-node deployment.
This is by all means the most common production setup, in which each organization has their own infrastructure setup in their own datacenter.
Not surprisingly there are many attempts to build such a setup, and currently I see more pre-built, pre-configured and more user-friendly tools (e.
g.
Amazon Managed Blockchain or FabDep) such that deploying multiple-node is not as difficult as expected.
Nevertheless, for those who wish to become master in Hyperledger Fabric, I still suggest a more hands-on practice on the whole process.
This maintains the deployment flexibility, and most importantly, helps one understand more about the Hyperledger Fabric architecture.
One challenge of building multi-node deployment is the communication between docker containers.
In a localhost everything can talk to each other.
In real life it never happens.
When I first research for those successfully bringing up multi-node setup, some of them relying on Docker Swarm (e.
g, link).
It is of no surprise as Hyperledger Fabric uses docker containers.
But the deployment with Docker Swarm always gets into challenges, if not mentioning many concerns about Docker Swarm.
There is a simpler way from λ.
eranga.
by using extra hosts.
This makes the communication between running containers easier, provided that we are feeding the necessary information when bringing up the network.
Inspired by this work, here I am deploying a three-organization fabric network running on three separate machines.
And what makes more fun is I will deploy two channels on this setup, and see how things are working.
SetupIt is a very simple setup.
In this fabric network we only have one orderer.
There are total three organizations, in each of which a peer node is configured.
For simplicity we take out most of components, and only observe the behaviour of setting up channels in multiple nodes.
We will first build a channel called channelall.
All three organizations (peers) join channelall.
After creation of channelall, we will install and instantiate the chaincode Simple Asset Chaincode (sacc) on channelall, and observe the behaviour of chaincode query and invocation across the three nodes.
Later we add one more channel channel12, in which only Org1 and Org2 join.
We will again install and instantiate the sacc chaincode on channel12, and observe the behaviour of chaincode query and invocation for non-member and for cross-channel perspective.
Here is how the setup looks like.
Demo StepsThe process in this demo follows what we learn from Build Your First Network (BYFN).
What’s new is to crate four nodes on AWS.
Step 1: Bring Up the Four NodesStep 1.
1: Create 4 AWS EC2 instancesFollow standard creation of EC2 instances.
Note that,Select Ubuntu 16.
04 LTSUse t2.
small instancesFor demo purpose, just use Security Group that open everything.
In production, only open those IP and ports which is neededWe will designate the EC2 instances as followingOrdererNode 1 for Org1Node 2 for Org2Node 3 for Org3The four EC2 instances for demonstration.
Step 1.
2: Install prerequisite and Hyperledger Fabric related materialOn each node, install the prerequisite (link) and Hyperledger Fabric related material (link), including docker images, the tools and fabric samples.
After installation use Build Your First Network (BYFN) and see if the whole process completes.
$ cd fabric-samples$ .
/byfn.
sh upAnd bring down the BYFN with$ .
/byfn.
sh downHere I write down the Private IP and Public IP on all four nodes, and I will use these during the whole article.
When you bring up your machines, make sure you are using your own set of addresses.
It is only the IP addresses of my EC2 instances.
Use your own in your setup.
Step 2: Generate Crypto and Network ArtifactsAs a practice of Build Your First Network (BYFN), we first prepare configurations.
In this demo I use my own localhost.
First, prepare a project folder.
I create it inside fabric-samples.
It is not a must, but we will refer some directories later.
If you are using another directory, make sure you can refer the right contents.
In my localhost,$ cd fabric-samples$ mkdir 3node2channel && cd 3node2channelWe need to create a directory for channel-artifacts and deployment.
The directory crypto-config is created automatically when running the tool later.
$ mkdir channel-artifacts$ mkdir deploymentAt this point we work on the two files: crypto-config.
yaml and configtx.
yaml.
Later we will build channels using profile ChannelAll and Channel12.
Here is how the directory looks like.
Now we can generate the required artifacts.
First, crypto artifacts$ .
/bin/cryptogen generate — config=.
/crypto-config.
yamlNow, the network artifacts.
First we work on the genesis block for orderer$ export FABRIC_CFG_PATH=$PWD$ .
/bin/configtxgen -profile ThreeOrgsOrdererGenesis -outputBlock .
/channel-artifacts/genesis.
blockWe also need to configuration transaction for the both channels.
$ .
/bin/configtxgen -profile ChannelAll -outputCreateChannelTx .
/channel-artifacts/channelall.
tx -channelID channelall$ .
/bin/configtxgen -profile Channel12 -outputCreateChannelTx .
/channel-artifacts/channel12.
tx -channelID channel12After the generation we now have the artifacts required for our setup.
Step 3: Examine the Docker-Compose FilesAnother set of important files are the docker compose files, which determine what and how nodes are brought up and running for the setup.
They are all stored under deployment directory.
The structure we have is like this.
docker-compose-base.
yml: This is the base for both orderers and peers.
This file is referred in other docker compose files.
docker-compose-orderer.
yml: This is for orderer node.
docker-compose-node#.
yml: This is for node of Org1, Org2 and Org3.
There are some remarks on these files.
As said, the directory structure has been considered.
If you are not following the previous steps, make sure you update the directory (mainly on the volumes)There are entries of extra hosts (extra_hosts).
It is quite straight forward: what we need is to specify other nodes with their (private) IP addresses.
These entries will then be configured inside /etc/hosts of the containers.
Each Org comes with a peer and a CLI.
We will use CLI for chaincode operations.
Here are the directory structure.
Step 4: Upload the 3node2channel Directory to All NodesIn my local host// fabric-samples directory$ tar cf 3node2channel.
tar 3node2channel/// update to all nodes (use your own key file and public IP addresses)$ scp -i ~/Downloads/aws.
pem 3node2channel.
tar ubuntu@3.
90.
64.
249:/home/ubuntu/fabric-samples/$ scp -i ~/Downloads/aws.
pem 3node2channel.
tar ubuntu@3.
92.
233.
164:/home/ubuntu/fabric-samples/$ scp -i ~/Downloads/aws.
pem 3node2channel.
tar ubuntu@18.
233.
151.
196:/home/ubuntu/fabric-samples/$ scp -i ~/Downloads/aws.
pem 3node2channel.
tar ubuntu@52.
23.
207.
17:/home/ubuntu/fabric-samples/My localhostStep 5: Extract the Directory and Bring Up NodesFor all nodes// in fabric-samples$ tar xf 3node2channel$ cd 3node2channel/deploymentFor Orderer node$ docker-compose -f docker-compose-orderer.
yml up -d$ docker psOrderer NodeFor Org1 node$ docker-compose -f docker-compose-node1.
yml up -d$ docker psNode 1 for Org1For Org2 node$ docker-compose -f docker-compose-node2.
yml up -d$ docker psNode 2 for Org2For Org3 node$ docker-compose -f docker-compose-node3.
yml up -d$ docker psNode 3 for Org3Step 6: Setup Channel channelallHere we take Node 1 to create the genesis block for channel channelall.
$ docker exec -e “CORE_PEER_MSPCONFIGPATH=/var/hyperledger/users/Admin@org1.
example.
com/msp” peer0.
org1.
example.
com peer channel create -o orderer.
example.
com:7050 -c channelall -f /var/hyperledger/configs/channelall.
txThe file channelall.
block is created.
Node 1We will first let peer0.
org1.
example.
com join channelall first using this channelall.
block file.
$ docker exec -e “CORE_PEER_MSPCONFIGPATH=/var/hyperledger/users/Admin@org1.
example.
com/msp” peer0.
org1.
example.
com peer channel join -b channelall.
blockNode 1: peer0.
org1 joins channelallWe have peer0.
org1.
example.
com now in channelall.
We will do the same on peer0.
org2.
example.
com and peer0.
org3.
example.
com.
Note that the file is now inside container peer0.
org1.
example.
com on node 1.
We will use localhost as “bridge” to have this file sent to node 2 and 3, and then passed to peer0.
org2.
example.
com and peer0.
org3.
example.
com, respectively.
Here are the steps (again, update the public IP addresses of your own instances):// node1$ docker cp peer0.
org1.
example.
com:channelall.
block .
// localhost$ scp -i ~/Downloads/aws.
pem ubuntu@3.
92.
233.
164:/home/ubuntu/fabric-samples/3node2channel/deployment/channelall.
block .
$ scp -i ~/Downloads/aws.
pem channelall.
block ubuntu@18.
233.
151.
196:/home/ubuntu/fabric-samples/3node2channel/deployment/$ scp -i ~/Downloads/aws.
pem channelall.
block ubuntu@52.
23.
207.
17:/home/ubuntu/fabric-samples/3node2channel/deployment/// node2$ docker cp channelall.
block peer0.
org2.
example.
com:/channelall.
block// node3$ docker cp channelall.
block peer0.
org3.
example.
com:/channelall.
blockNode 1: copy the channelall.
block from peer0.
org1 to Node 1My localhost: scp the channelall.
block from Node 1 and to Node 2 and 3Node 2: copy the channelall.
block from Node 2 to peer0.
org2Node 3: copy the channelall.
block from Node 3 to peer0.
org3We see the file channelall.
block is now in both Node 2 and Node 3.
Finally we can have peer0.
org2.
example.
com and peer0.
org3.
example.
com joining channelall.
Node 2$ docker exec -e “CORE_PEER_MSPCONFIGPATH=/var/hyperledger/users/Admin@org2.
example.
com/msp” peer0.
org2.
example.
com peer channel join -b channelall.
blockNode 2: peer0.
org2 joins channelallNode 3$ docker exec -e “CORE_PEER_MSPCONFIGPATH=/var/hyperledger/users/Admin@org3.
example.
com/msp” peer0.
org3.
example.
com peer channel join -b channelall.
blockNode 3: peer0.
org3 joins channelallNow all three peers in three organization have joint channelall.
Step 7: Chaincode OperationsWe are using Simple Asset Chaincode (sacc), which is already inside fabric-sample.
In all nodes, we install the sacc chaincode.
$ docker exec -it cli peer chaincode install -n mycc -p github.
com/chaincode/sacc -v v0Chaincode Installation in All nodes: here Node 1 is shownAfter chaincode installation, we will instantiate the chaincode on one of the node.
In the instantiation we specify an initial value (key: a, value: 100).
We also specify endorsement policy only requires one of the three organizations for sake of simplicity.
Node 1$ docker exec -it cli peer chaincode instantiate -o orderer.
example.
com:7050 -C channelall -n mycc github.
com/chaincode/sacc -v v0 -c ‘{“Args”: [“a”, “100”]}’ -P “OR(‘Org1MSP.
member’, ‘Org2MSP.
member’,’Org3MSP.
member’)”Chaincode Instantiation with “a” set to 100 on Node 1Now we can query from other nodes, say Node 2 for the value of a.
Node 2$ docker exec -it cli peer chaincode query -C channelall -n mycc -c ‘{“Args”:[“query”,”a”]}’Query value of “a” on Node 2And we see 100.
And same result if we query from Node 1 and Node 3.
We now invoke the chaincode and set the value of a to 200.
Here we do it on Node 3.
Node 3$ docker exec -it cli peer chaincode invoke -o orderer.
example.
com:7050 -C channelall -n mycc -c ‘{“Args”:[“set”,”a”, “200”]}’Invoke to set value of “a” to 200 on Node 3Again again, we check the value on Node 2Node 2$ docker exec -it cli peer chaincode query -C channelall -n mycc -c ‘{“Args”:[“query”,”a”]}’Query the value of “a” again on Node 2We now see 200.
And same result if we query from Node 1 and Node 3.
So our fabric network across three nodes and three organizations works well on channelall.
Step 8: Setup Channel channel12 and Chaincode InstantiationThis is same as Step 6 and 7, but now we are working on channel12.
I omit those screenshot as it’s almost the same as in Step 6.
Here we take Node 1 to create the genesis block for channel channel12.
$ docker exec -e “CORE_PEER_MSPCONFIGPATH=/var/hyperledger/users/Admin@org1.
example.
com/msp” peer0.
org1.
example.
com peer channel create -o orderer.
example.
com:7050 -c channel12 -f /var/hyperledger/configs/channel12.
txThe file channel12.
block is created.
We will first let peer0.
org1.
example.
com join channel12 first using this channel12.
block file.
$ docker exec -e “CORE_PEER_MSPCONFIGPATH=/var/hyperledger/users/Admin@org1.
example.
com/msp” peer0.
org1.
example.
com peer channel join -b channel12.
blockWe have peer0.
org1.
example.
com now in channel12.
We will do the same on peer0.
org2.
example.
com.
Here are the steps (again, update the public IP addresses of your own instances):// node1$ docker cp peer0.
org1.
example.
com:channel12.
block .
// localhost$ scp -i ~/Downloads/aws.
pem ubuntu@3.
92.
233.
164:/home/ubuntu/fabric-samples/3node2channel/deployment/channel12.
block .
$ scp -i ~/Downloads/aws.
pem channel12.
block ubuntu@18.
233.
151.
196:/home/ubuntu/fabric-samples/3node2channel/deployment/// node2$ docker cp channel12.
block peer0.
org2.
example.
com:/channel12.
blockFinally we can have peer0.
org2.
example.
com joining channel12.
Node 2$ docker exec -e “CORE_PEER_MSPCONFIGPATH=/var/hyperledger/users/Admin@org2.
example.
com/msp” peer0.
org2.
example.
com peer channel join -b channel12.
blockNow Org1 and Org2 have joint channel12.
We are using sacc as the chaincode demonstration, they are already installed in the peers.
What we need is to instantiate the chaincode on channel12.
For sake of demonstration, we specify another initial value (key: b, value: 1).
Node 1$ docker exec -it cli peer chaincode instantiate -o orderer.
example.
com:7050 -C channel12 -n mycc github.
com/chaincode/sacc -v v0 -c ‘{“Args”: [“b”, “1”]}’ -P “OR(‘Org1MSP.
member’, ‘Org2MSP.
member’)”Chaincode Instantiation with “b” set to 1 on Node 1, on channel12Query from Node 2, and we see the correct result.
$ docker exec -it cli peer chaincode query -C channel12 -n mycc -c ‘{“Args”:[“query”,”b”]}’Query value of “b” from Node 2Query from Node 3, and we see error as Node 3 (Org 3) is not in channel12.
Query value of “b” from Node 3: Org3 is not in channel12Finally we will check whether we can get the key/value in different channels.
In Node 1, we try to get the value of b on channelall.
$ docker exec -it cli peer chaincode query -C channelall -n mycc -c ‘{“Args”:[“query”,”b”]}’Query value of “b” from chanelall on Node 1As we can see, asset b is not defined in the ledger of channelall.
Each channel has its own ledger.
Even a peer joins multiple channels the ledger states of these channels are not shared.
Step 9 Clean UpWhen we complete our setup, we can clean up everything.
On each node, use docker-compose to tear down the containers.
We also clean up the images created during chaincode operation.
// orderer$ docker-compose -f docker-compose-orderer.
yml down// node1$ docker-compose -f docker-compose-node1.
yml down$ docker rm $(docker ps -aq)$ docker rmi $(docker images net-* -q)// node2$ docker-compose -f docker-compose-node2.
yml down$ docker rm $(docker ps -aq)$ docker rmi $(docker images net-* -q)// node3$ docker-compose -f docker-compose-node3.
yml down$ docker rm $(docker ps -aq)$ docker rmi $(docker images net-* -q)You can determine whether you keep the 3node2channel tar file and directory.
Also you can determine whether you stop the AWS EC2 instances (for next demo) or terminate them.
SummaryIn this article we show how to deploy a three-node setup for three organizations.
Although it seems a bit complicated, the logic is quite straight forward.
We largely use what we learn from Build Your First Network (BYFN), and with careful preparation of the docker compose files, we demonstrate how multiple channels are created and observe the behaviour of same chaincode across multiple channels.
.