Writing a load testing tool in GoAniket GadreBlockedUnblockFollowFollowingJun 13Motivation :In one of my previous organizations we had a requirement of load testing Python and Scala based micro-services which interacted over RabbitMQ (using protocol AMQP 0.
9 which is pretty different from AMQP 1.
0)We were already using Jmeter as our preferred load testing tool for Rest APIs.
So I had few options to work withJmeter with some AMQP based pluginRabbitMQ itself provides a crude way of loading queues via its PerfTest toolLast option was to build something of my ownJmeter and RabbitMQ tool could have served the purpose to some extent but they were more suitable for async messages.
Some of our micro-services used rpc (Remote Procedure Call) functionality of RabbitMQ which is more of a synchronous communication between Consumer and Publisher over queues.
Now whether the usage of RPC over message queues was a correct design choice or not is a topic of discussion for some other time :)Conclusion was to not use Jmeter plugin or RabbitMQ Perf Test tool but to build something of my ownI was learning Go around that time and was super impressed with Go’s concurrency model.
I thought it was a good time and opportunity to convert my Go learning into implementing something useful so I decided to build load generator tool in Go.
BTW there is a nice load generator tool written in Go called Vegeta but it’s mainly for HTTPBefore starting to build the tool, I wanted to confirm few thingsThere was a RabbitMQ client library available in Go because RabbitMQ doesn’t provide one, fortunately there was one open-source client libraryMonitoring RabbitMQ in terms of CPU, Memory, Disk IO of host that runs the RabbitMQ node and then the metrics of individual queues itself like total connections, total consumers, total messages etc.
at given point of time.
There are multiple ways to do this as documented hereDesign :Once the above things were confirmed, I started thinking about the bare minimum functionality this tool should have and I decided to follow Jmeter’s design for the same.
I needed to controlNo of Threads/UsersRamp up time for this usersExecution time of testWay to define samplers/testsThat’s it.
If I could manage these 4 things my tool should be ready for use.
Code:First thing was to accept user input for No of User, Ramp Up time and Execution time.
This was pretty straight forward, I could set it with command line flagsGo encourages having shorter variable names.
If anyone is more interested in knowing more, here is one presentationSecond thing was to calculate how much time should Go wait before it starts next user thread, which was also straightforward as I had the total no.
of users to load and time in which all user threads should be up and running, so wait time in between initiation of two consecutive user threads wasSo for example if my ramp up time was 30 secs and I had total of 10 user I would spin off a new user thread every 3 secs.
Now I wanted something which could increment the user thread count as per the ramp up calculations.
Best way was to initiate a goroutine for that job.
Goroutine is a lightweight thread created by Go’s runtime when any named or anonymous function is invoked with a prefixed keyword goBelow goroutine pushes user count into the channel every time its incremented but won’t initiate the actual user thread.
Channel object in Go is a communication medium for goroutines to interact with each other by sending and receiving messages and is defined by keyword chanNext thing was to get a time channel which would help in indicating the end of execution durationAfter this the final bit was to use the two channels defined in above code for controlling the flow and executing the required testsThis piece of code brings together all the core features of Go which makes concurrency so easyselect is a special statement in Go which blocks the execution until any one of the channels in case statements is ready for send/receive operationAnd that was all I needed !!.I just added all required test calls in the above for loop and was able to bombard RabbitMQ message queues with hundreds of users that to with extremely low memory footprint on my load generator machine !For monitoring and reporting I pushed all metrics into InfluxDB and used Grafana for real time visualizationGo is an extremely powerful language when it comes to concurrency.
I would recommend every Go enthusiast to read the book Concurrency in Go written Katherine Cox-BudayYou can find the full gist of the above code here.