Generating Porter2 FSM For Fun and Performance in Go

GoDoc

tl;dr

  • This post describes the Porter2 package I implemented. It is written in Go (#golang).
  • By using a finite-state-machine approach to Porter2 stemming, I was able to achieve 660% better performance compare to other Go implementations.
  • FSM-based approach is great for known/fixed data set, but obviously not workable if the data set changes at runtime.
  • Hand-coding FSM is a PITA!!! Automate if possible.

Introduction

In a personal project I am working on, I had the need to perform word stemming in two scenarios. First, I need to perform stemming for all the string literals in a LARGE corpus and then determine if the words are in a fixed set of literals. Second, I need to perform stemming for a subset of words in real-time, as messages stream in.

In the first case, performance is important but not critical; in the second case, performance is a huge factor.

Stemming

To start, according to wikipedia:

Stemming is the term used in linguistic morphology and information retrieval to describe the process for reducing inflected (or sometimes derived) words to their word stem, base or root form—generally a written word form.

As a quick example, the words fail, failed, and failing all mean something has failed. By stemming these three words, I will get a single form which is fail. I can then just use fail going forward instead of having to compare all three forms all the time.

The Porter stemming algorithm is by far the most commonly used stemmer and also considered to be one of the most gentle stemmers. The Porter stemming algorithm (or ‘Porter stemmer’) works by removing the commoner morphological and inflexional endings from words in English. Its main use is as part of a term normalisation process that is usually done when setting up Information Retrieval systems. (ref)

Porter2 is universally considered to be an enhancement over the original Porter algorithm. Porter2 has an improved set of rules and it’s widely used as well.

Implementation

This package, Porter2, implements the Porter2 stemmer. It is written completely using finite state machines to perform suffix comparison, rather than the usual string-based or tree-based approaches. As a result, it is 660% faster compare to string comparison-based approach written in the same (Go) language.

This implementation has been successfully validated with the dataset from http://snowball.tartarus.org/algorithms/english/, so it should be in a usable state. If you encounter any issues, please feel free to open an issue.

Usage is fairly simple:

import "github.com/surgebase/porter2"

fmt.Println(porter2.Stem("seaweed")) // should get seawe

Performance

This implementation by far has the highest performance of the various Go-based implementations, AFAICT. I tested a few of the implementations and the results are below.

Implementation Time Algorithm
surgebase 319.009358ms Porter2
dchest 2.106912401s Porter2
kljensen 5.725917198s Porter2

To run the test again, you can run compare.go (go run compare.go).

State Machines

Most of the implementations, like the ones in the table above, rely completely on suffix string comparison. Basically there’s a list of suffixes, and the code will loop through the list to see if there’s a match. Given most of the time you are looking for the longest match, so you order the list so the longest is the first one. So if you are luckly, the match will be early on the list. But regardless that’s a huge performance hit.

This implementation is based completely on finite state machines to perform suffix comparison. You compare each chacter of the string starting at the last character going backwards. The state machines will determine what the longest suffix is.

As an example, let’s look at the 3 suffixes from step0 of the porte2 algorithm. The goal, and it’s the same for all the other steps, it’s to find the longest matching suffix.

'
's
's'

If you were to build a non-space-optimized suffix tree, you would get this, where R is the root of the tree, and any node with * is designated as a final state:

        R
       / \
      '*  s
     /     \
    s       '*
   /
  '*

This is a fairly easy tree to build, and we actually did that in the FSM generator we will talk about later. However, to build a working suffix tree in Go, we would need to use a map[rune]*node structure at each of the nodes. And then search the map for each rune we encounter.

To test the performance of using a switch statement vs using a map, I wrote a quick test:

switch: 4.956523ms
   map: 10.016601ms

The test basically runs a switch statement and a map each for 1,000,000 times. So it seems like using a switch statement is faster than a map. Though I think the compiler basically builds a map for all the switch case statements. (Maybe we should call this post Microbenchmarking for fun and performance?)

In any case, let’s go with the switch approach. We basically need to unroll the suffix tree into a finite state machine.

        R0
       / \
      '1* s2
     /     \
    s3      '4*
   /
  '5*

To do that, we need to assign a state number to each of the nodes in the suffix tree, and output each of the states and the transitions based on the rune encountered. The tree above is the same as the one before, but now has a state number assigned to each node.

Generator

I actually started building all the porter2 FSMs manually with a completely different approach than what I am describing here. I won’t go into details here but needless to say, it was disastrous. Not only was hand coding state machines extremely error-prone, the approach I was taking also had a lot of potential for bugs. It took me MANY HOURS to hand build those FSMs but at the end, I was happy to abandon all of them for the approach I am taking now.

To reduce errors and make updating the FSM easier, I wrote a quick tool called suffixfsm to generate the FSMs. The tool basically takes a list of suffixes, creates a suffix tree as described above, and unrolls the tree into a set of states using the switch statement.

It took me just a couple hours to write and debug the tool, and I was well on my way to fixing other bugs!

For example, running the command go run suffixfsm.go step0.txt generated the following code. This is a complete function for step0 of the porter2 algorithm. The only thing missing is what to do with each of the final states, which are in the last switch statement.

var (
		l int = len(rs) // string length
		m int			// suffix length
		s int			// state
		f int			// end state of longgest suffix
		r rune			// current rune
	)

loop:
	for i := 0; i < l; i++ {
		r = rs[l-i-1]

		switch s {
		case 0:
			switch r {
			case '\'':
				s = 1
				m = 1
				f = 1
				// ' - final
			case 's':
				s = 2
			default:
				break loop
			}
		case 1:
			switch r {
			case 's':
				s = 4
			default:
				break loop
			}
		case 2:
			switch r {
			case '\'':
				s = 3
				m = 2
				f = 3
				// 's - final
			default:
				break loop
			}
		case 4:
			switch r {
			case '\'':
				s = 5
				m = 3
				f = 5
				// 's' - final
			default:
				break loop
			}
		default:
			break loop
		}
	}

	switch f {
	case 1:
		// ' - final

	case 3:
		// 's - final

	case 5:
		// 's' - final

	}

	return rs

Finally

This is a technique that can probably be applied to any fixed data set. The performance may vary based on the size of the state machine so test it with both maps and FSM to see what works best.

Happy Go’ing!

Go: From a Non-Programmer's Perspective

Warning: Long Post. Over 3900 words according to wc. So read at your own risk. :)

Go is a fairly recent programming language created by Robert Griesemer, Rob Pike and Ken Thompson of Google. It has risen in popularity over the the past few years, especially since Go 1.0 was released.

There are a ton of posts out there that talks about the pros and cons of Go, and why one would use it or not. In addition, there’s a bunch of posts out there written by different developers coming from different perspectives, such as Python, Ruby, Node, Rust, etc, etc. Recently I even read a couple of Chinese blog posts on why Go is popular in China, and why some of the Chinese developers have abandoned Go, which are quite interesting as well.

This post is my perspective of Go, how I picked it up, and what I think of it after using it for a while. It is not a post about why Go is better or worse than other languages.

In short, I like Go. It’s the first programming language I’ve used in recent years that I can actually build some interesting projects, e.g., SurgeMQ (detailed post), in my limited spare time.

My Background

I am not a programmer/developer. Not full-time, not part-time, not moonlight. I tell my colleagues and teams that “I am not technical.”

But I do have a technical background. I have a MSCS degree from way back when, and have spent the first 6-7 years of my career performing security audits and penetration tests, and building one of the world’s largest managed security services (at least at the time).

My programming langauge progression, when I was technical, has been BASIC (high school), Pascal and C (college), Perl, PHP, Java, and Javascript (during my technical career). I can’t claim to be an “expert” in any of these languages, but I consider myself quite proficient in each at the time I was using them.

I was also reasonably network and system savvy, in the sense that I can get myself in and around the Solaris and Linux (UN*X) systems pretty well, and understand the networking stack sufficiently. I consider myself fairly proficient with the various system commands and tools.

For the past 12 years, however, I have not been a developer, nor a systems guy, nor a networking guy. Instead, I have been running product management for various startups and large companies in the security and infrastructure space.

Since the career change, I’ve not done any meaningful code development. I’ve written a script here and there, but nothing that I would consider to be “software.” However, I’ve managed engineering teams as part of my resonsibility, in addition to product management, to produce large scale software.

In the past 12 years, my most used IDE is called Microsoft Office. So, in short, I am probably semi-technical, and know just enough to be dangerous.

My History with Go

In 20112012, I had the responsibility of building a brand new engineering team (I was already running product management) at VMware to embark on a new strategic initiative. The nature of the product/service is not important now. However, at the time, because the team is brand new, we had some leeway in choosing a language for the project. VMware at the time was heavily Java, and specifically Spring given the 2009 acquisition of SpringSource. While the new team had mostly Java experience, there was desire to choose something less bloated, and something that had good support for the emerging patterns of distributed software.

First Touch

Some of the team members had experience with Scala, so that became an obvious option. I did some research on the web, and found some discussions of Go. At the time, Go hasn’t reached 1.0 yet, but there was already a buzz around it. I looked on Amazon, and found The Way to Go, which was probably the only Go book around at the time. For $3 on the Kindle, it was well worth it. However, due to the nascent nature of Go (pre 1.0), it was not a comfortable choice so I didn’t put that as an option. But this was my first touch of Go and it felt relatively painless.

At the end, the team chose Scala because of existing experience, and that in theory, people with Java experience should move fairly easily to Scala. We were the first team in VMware to use Scala and we were pretty excited about it.

However, to this date, I am still not sure we made the right decision to move to Scala (not that it’s wrong either.) The learning curve I believe was higher than we originally anticipated. Many of the developers wrote Java code w/ Scala syntax. And hiring also became an issue. Basically every new developer that came onboard must be sent to Typesafe for training. It was simply not easy for most developers who came from a non-functional mindset to jump into a totally functional mindset. Lastly, the knowledge differences of new Scala developers and experienced ones made it more difficult for them to collaborate.

I also tried to read up Scala and at least understand the concept. I even tried to take the online course on Coursera offered by Martin Odersky. However, I just could not get my non-functional mind to wrap around the functional Scala. And since I really didn’t need to code (nor the developers want me to), I gave up on learning Scala.

Second Touch

In any case, fast forward 2 years to Q3 of 2013. I had since left VMware and joined my current company, Jolata, to build a big data network analytics solutions for mobile carriers and high-frequency trading financial services firms. We are a small startup that’s trying to do a ton of things. So even though I run products, I have to get my hands dirty often.

One of the things we had to do as a company is to build a repeatable demo environment. The goal is to have a prebuilt vagrant VM that we can run on our Macs, and we can demonstrate our product without connecting to the network. The requirement was that we had an interesting set of data in the database so we can walk through different scenarios.

The data set we needed was network flow data. And to make the UI look realistic, interesting and non-blocky, we wanted to generate noisy data so the UI looks like it’s monitoring a real network. Because all of the developers are focused on feature development, I took on the task of building out the data set.

By now, Go has released v1.1 and on its way to 1.2. It was then I started seriously considering Go as a candidate for this project. To build a tool that can generate the data set, we needed two libraries. The first is a Perlin Noise generator, and the second is Google’s Cityhash. Neither of these were available in Go (or not that I remember). I thought this would be a great opportunity to test out Go. The end results were my Go Learn Projects #0 Perlin, and #1 Cityhash.

Both of these projects were relatively simple since I didn’t have to spend a lot of time figuring out HOW to write them. Perlin Noise has well-established C libraries and algorithms, and Cityhash was written in C so it was easy to translate to Go. However, these projects gave me a good feel of how Go works.

In the end, I wrote the data generator in Go (private repo) and got the first taste of goroutines. Again, this second touch with Go was also relatively painless. The only confusion I had at the time was the Go source tree structure. Trying to understand $GOROOT, $GOPATH and other Go environment variables were all new to me. This was also the first time in 10 years that I really spent time writing a piece of software, so I just considered the confusion as my inexperience.

Third Touch and Beyond

Today, I no longer code at work as we have more developers now. Also, the Jolata product is mostly C/C++, Java and Node, so Go is also no longer in the mix. However, After getting a taste of Go in the first couple of Go projects, I’ve since spent a tremendous amount of my limited personal spare time working with it.

I have since written various libraries for bitmap compression, integer compression, bloom filters, skiplist, and many others. And I have blogged my journey along the way as I learn. With these projects, I’ve learned how to use the Go toolchain, how to write idiomatic Go, how to write tests with Go, and more importantly, how to optimize Go.

Interestingly, one of my most popular posts is Go vs Java: Decoding Billions of Integers Per Second. This tells me that a lot of Java developers are potentially looking to adopt Go.

All these have allowed me to learn Go enough to build a real project, SurgeMQ. It is by far my most popular project and one that I expect to continue developing.

My Views on Go

Go is not just a langauge, it also has a very active community around it. The views are based on my observation over the past 1.5 years of using Go. My Go environment is primary Sublime Text 3 with GoSublime plugin.

As a Language…

I am not a language theorist, nor do I claim to be a language expert. In fact, prior to actually using Go, I’ve barely heard of generics, communicating sequential processes, and other “cool” and “advanced” concepts. I’ve heard of all the new cool programming languages such as Clojure and Rust, but have never looked at any of the code. So my view of Go is basically one of a developer n00b.

In a way, I consider that to be an advantage coming in to a new programming language, in that I have no preconceived notion of how things “SHOULD” be. I can learn the language and use the constructs as they were intended, and not have to question WHY it was designed that way because it’s different than what I know.

Others may consider this to be a huge disadvantage, since I don’t know any better. There maybe constructs in other languages that would make my work a lot easier, or make my code a lot simpler.

However, as long as the language doesn’t slow me down, then I feel it’s serving my needs.

Go is Simple

As a language for a new deverloper, Go was very easy to pick up. Go’s design is fairly simple and minimalistic. You can sit down and read through the Language Specification fairly quickly in an idle afternoon. I actually didn’t find the language reference until later. My first touch of Go was by scanning through the book The Way To Go. Regardless, there’s not a lot to the language so it’s relatively easy for someone like myself to pick up the basics. (Btw, I’ve also never gone through the Go Tour. I know it’s highly recommended to all new Go developers. I just never did it.)

There are more advanced concepts in Go, such as interface, channel, and goroutine. Channel in general is a fairly straightforward concept. Most new programmers should be able to understand that quickly. You write stuff in, you read stuff out. It’s that simple. From there, you can slowly expand on the concept as you go along by adding buffered channels, or ranging over channels, or checking if the read is ok, or using quit channels.

For anyone coming from a language with threads, goroutine is not a difficult concept to understand. It’s basically a light-weight thread that can be executed concurrently. You can run any function as a goroutine.

The more difficult concept is interface. That’s because it’s a fairly new concept that doesn’t really exist in concept that’s fairly different than other languages. Once you understand what interfaces are, it’s fairly easy to start using them. However, designing your own interfaces is a different story.

The one thing I’ve seen most developers complain about Go is the lack of generics. Egon made a nice Summary of Go Generics Discussions that you can read through. For me personally, I don’t know any better. I have never used generics and I haven’t found a situation where I strongly require it.

As a language a team, the simplicity of Go is HUGE. It allows develoeprs to quickly come up to speed and be productive in the shortest period of time. And in this case, time is literally money.

Go is Opinionated

Go is opinionated in many ways. For example, probably one of the most frustrating thing about Go is how to structure the code directory. Unlike other languages where you can just create a directory and get started, Go wants you to put things in $GOPATH. It took a few readings of How to Write Go Code for me to grasp what’s going on, and it took even longer for me to really get the hang of code organization, and how Go imports packages (e.g., go get).

If I go back and look at my first internal project, I would probably cry because it’s all organized in a non-idiomatic way. However, once I got the hang of how Go expects things to be organized, it no longer was a obstacle for me. Instead of fighting the way things should be organized in Go, I learned to go with the flow. At the end of the day, the $GOPATH organizational structure actually helps me track the different packages I import.

Another way Go is opinionated is code formatting. Go, and Go developers, expect that all Go programs are formatted with go fmt. A lot of developers hate it and some even listed it as a top reason for leaving Go. However, this is one of those things that you just have to learn to go with the flow. Personally I love it.

And as a team language it will save a ton of argument time. Again, time is money for a new team. When my new VMware team got started, we probably spent a good 30 person-hours debating code formatting. That’s $2700 at a $180K fully-burdened rate. And that’s not counting all the issues we will run into later trying to reformat code that’s not properly formatted.

Go is also very opininated in terms of variable use and package import. If a variable is declared but not used, the Go compiler will complain. If a package is imported but not used, the Go compiler will complain. Personally, I like the compiler complaining about the unused variables. It keeps the code clean, and reduce the chance of unexpected bugs. I am less concerned about unused packages but have also learned to live with the compiler complains. I use goimports in Sublime Text 3 to effectively and quickly take care of the import statements. In fact, in 99% of the cases I don’t even need to type in the import statements myself.

Go is Safe

Go is safe for a couple of reasons. For a new developer, Go does not make it easy for you to be lazy. For example, Go is a statically typed language, which means every variable must explicitly have a type associated with it. The Go compiler does infer types under certain situations, but regardless, there’s a type for every variable. This may feel uncomfortable for developers coming from dynamic languages, but the benefit definitely outweighs the cost. I’ve experience first hand, as a product person waiting for bugs to be fixed, how long it takes to troubleshoot problems in Node. Having static types gives you a feeling of “correctness” after you have written the code.

Another example of Go not allowing you to be lazy is that Go’s error handling is through the return of error from functions. There has been a ton of discussions and debates on the merit of error vs exception handling so I won’t go through it here. However, for a new programmer, it really requires your explicit attention to handle the errors. And I consider that to be a good thing as you know what to expect at each step of the program.

Making things explicit and making it harder for developers to be lazy are a couple of the reasons that make Go safe.

Another reason is that Go has a garbage collector. This makes it different from C/C++ as it no longer require developers to perform memory management. The difficulty in memory management is the single biggest source of memory leaks in C/C++ programs. Having a GC removes that burden from developers and makes the overall program much safer. Having said that, there’s much improvement to be made to the GC given its nascent state. And, as I learned over the past 1.5 years, to write high performance programs in Go today, developers need to make serious efforts to reduce GC pressure.

Again, as a team langauge, the safety aspect is very important. The team will likely end up spending much less time dealing with memory bugs and focus more on feature development.

Go is Powerful

What makes Go powerful are its simplicity, its high performance, and advanced concepts such as channels, goroutines, interfaces, type composition, etc. We have discussed all of these in previous sections.

In addition to all that, one of the killer feature of Go is that all Go programs are statically compiled into a single binary. There’s no shared libraries to worry about. There’s no jar files to worry about. There’s no packages to bundle. It’s just a single binary. And that’s an extremely powerful feature from the deployment and maintenance perspectives. To deploy a Go program, you just need to copy a single Go binary over. To update it, copy a single Go binary over.

In contrast, to deploy a Node.js application, you may end up downloading hundreds of little packages at deployment time. And you have to worry about whether all these packages are compatible. The Node community has obviously developed a lot of good tools to manage dependencies and version control. But still, every time I see a Node app get deployed on a new machine, and have to download literally hundreds of little packages, I die a little inside.

Also, if you deploy C/C++ programs and depend on shared libraries, now you have to worry about OS and shared library version compatibility issues.

Another powerful feature of Go is that you can mix C and assembly code with Go code in a single program. I haven’t used this extensively, but in my attempt to optimize the integer compression library, I added different C and assembly snippets to try to squeeze the last ounce of performance out of Go. It was fairly easy and straightforward to do.

One last thing, Go has a very large and complete standard library. It enables developers to do most, if not all, of their work quickly and efficiently. As the language matures and the community grows, there will be more and more 3rd party open source libraries one can leverage.

As a Community

Today, Go has a very active community behind it. Specifically, the information sources I’ve followed and gotten help from include #go-nuts IRC, golang subreddit, and obviously the golang-nuts mailing list.

I spent quite a bit of time in IRC when I first started. I’ve gotten help from quite a few people such as dsal, tv42, and others, and I am grateful for that. I am spending less time there now because of the limited time I have (remember, my day job is not development. :)

There’s been some sentiments in the developer community that Go developers (gophers) are Google worshippers, don’t accept any feedbacks on language changes, harsh to new comers who come from different languages, difficult to ask questions because the sample code is not on play.golang.org, etc etc.

To be clear, I’ve never really spent much time with the different language communites, even when I was technical. So I have nothing else to compare to. So I can only speak from a human interaction level.

I can see it from both perspectives. For example, developers coming from different language backgrounds sometimes have experience with a different way of doing things. When they want to perfrom the same tasks in Go, they ask the question by saying here’s how I solved this problem in language X, how do I translate that to Go?

In some cases I’ve definitely seen people responding by saying that’s not how Go works and you are doing it wrong. That type of response can quickly create negative sentiment and kill the conversation.

Another type of response I’ve seen is some developers telling the original poster (OP) that they are not asking questions the right way, and then promptly sending the OP a link to some web page on how to properly ask questions. Again, I can see how the OP can have a negative view on the matter.

I’ve expereinced some of this myself. When I implemented a Bloom Filter package last year, I did a bunch of performance tests and wrote a blog post about the it. As a newbie learning Go, I felt like I accomplished something and I was pretty happy with it. I posted the link to reddit, and the first comment I got was

Downvoted because I dislike this pattern of learning a new language and then immediately publishing performance data about it, before you know how to write idiomatic or performant code in it.

Ouch!! As a new Go developer, this is not the response I expected. In the end though, the commenter also pointed out something that helped me improve the performance of the implementation. I was grateful for that. It was also then I realized how important it is to reduce the number of allocation in order to reduce the Go GC pressure.

In hindsight, the comment has a very valid point. I can understand why some developers would feel annoyed about benchmarks from people who have no idea on what they are doing. Regardless, being nice is not a bad thing. Saying things like “WTF is wrong with you” (not related to the bloom filter post) will only push new developers away.

I quickly got over the sting because I am just too old to care about what others think I should or should not do. I continued my learning process by writing and optimizing Go packages, and posting the results in my blog. In fact, the Go vs Java: Decoding Billions of Integers Per Second post has many of the optimization techniques I tried to increase the performance of Go programs.

Overall though, I felt I’ve learned a ton from the Go community. People have generally been helpful and are willing to offer solutions to problems. I have nothing to compare to, but I feel that the positives of the Go community far outweighs any negatives.

Conclusion

In conclusion, it has been a tremendous 1.5 years of working with Go, and seeing Go grow both as a language and as a community has been very rewarding.

My focus now, in my limited spare personal time, is to continue the development of SurgeMQ, which is a high performance MQTT broker and client library that aims to be fully compliant with MQTT 3.1 and 3.1.1 specs.

PingMQ: A SurgeMQ-based ICMP Monitoring Tool

pingmq is developed to demonstrate the different use cases one can use SurgeMQ, a high performance MQTT server and client library. In this simplified use case, a network administrator can setup server uptime monitoring system by periodically sending ICMP ECHO_REQUEST to all the IPs in their network, and send the results to SurgeMQ.

Then multiple clients can subscribe to results based on their different needs. For example, a client maybe only interested in any failed ping attempts, as that would indicate a host might be down. After a certain number of failures the client may then raise some type of flag to indicate host down.

There are three benefits of using SurgeMQ for this use case.

  • First, with all the different monitoring tools out there that wants to know if hosts are up or down, they can all now subscribe to a single source of information. They no longer need to write their own uptime tools.
  • Second, assuming there are 5 monitoring tools on the network that wants to ping each and every host, the small packets are going to congest the network. The company can save 80% on their uptime monitoring bandwidth by having a single tool that pings the hosts, and have the rest subscribe to the results.
  • Third/last, the company can enhance their security posture by placing tighter restrictions on their firewalls if there’s only a single host that can send ICMP ECHO_REQUESTS to all other hosts.

The following commands will run pingmq as a server, pinging the 8.8.8.0/28 CIDR block, and publishing the results to /ping/success/{ip} and /ping/failure/{ip} topics every 30 seconds. sudo is needed because we are using RAW sockets and that requires root privilege.

$ go build
$ sudo ./pingmq server -p 8.8.8.0/28 -i 30

The following command will run pingmq as a client, subscribing to /ping/failure/+ topic and receiving any failed ping attempts.

$ ./pingmq client -t /ping/failure/+
8.8.8.6: Request timed out for seq 1

The following command will run pingmq as a client, subscribing to /ping/failure/+ topic and receiving any failed ping attempts.

$ ./pingmq client -t /ping/success/+
8 bytes from 8.8.8.8: seq=1 ttl=56 tos=32 time=21.753711ms

One can also subscribe to a specific IP by using the following command.

$ ./pingmq client -t /ping/+/8.8.8.8
8 bytes from 8.8.8.8: seq=1 ttl=56 tos=32 time=21.753711ms

Commands

There are two builtin commands for pingmq.

pingmq server

Usage:
  pingmq server [flags]

 Available Flags:
  -h, --help=false: help for server
  -i, --interval=60: ping interval in seconds
  -p, --ping=[]: Comma separated list of IPv4 addresses to ping
  -q, --quiet=false: print out ping results
  -u, --uri="tcp://:5836": URI to run the server on

pingmq client

Usage:
  pingmq client [flags]

 Available Flags:
  -h, --help=false: help for client
  -s, --server="tcp://127.0.0.1:5836": PingMQ server to connect to
  -t, --topic=[]: Comma separated list of topics to subscribe to

IP Addresses

To list IPs you like to use with pingmq, you can use the following formats:

10.1.1.1      -> 10.1.1.1
10.1.1.1,2    -> 10.1.1.1, 10.1.1.2
10.1.1,2.1    -> 10.1.1.1, 10.1.2.1
10.1.1,2.1,2  -> 10.1.1.1, 10.1.1.2 10.1.2.1, 10.1.2.2
10.1.1.1-2    -> 10.1.1.1, 10.1.1.2
10.1.1.-2     -> 10.1.1.0, 10.1.1.1, 10.1.1.2
10.1.1.1-10   -> 10.1.1.1, 10.1.1.2 ... 10.1.1.10
10.1.1.1-     -> 10.1.1.1 ... 10.1.1.254, 10.1.1.255
10.1.1-3.1    -> 10.1.1.1, 10.1.2.1, 10.1.3.1
10.1-3.1-3.1  -> 10.1.1.1, 10.1.2.1, 10.1.3.1, 10.2.1.1, 10.2.2.1, 10.2.3.1, 10.3.1.1, 10.3.2.1, 10.3.3.1
10.1.1        -> 10.1.1.0, 10.1.1.1 ... 10.1.1.254, 10.1.1.255
10.1.1-2      -> 10.1.1.0, 10.1.1.1 ... 10.1.1.255, 10.1.2.0, 10.1.2.1 ... 10.1.2.255
10.1-2        -> 10.1.0.0, 10.1.0,1 ... 10.2.255.254, 10..2.255.255
10            -> 10.0.0.0 ... 10.255.255.255
10.1.1.2,3,4  -> 10.1.1.1, 10.1.1.2, 10.1.1.3, 10.1.1.4
10.1.1,2      -> 10.1.1.0, 10.1.1.1 ... 10.1.1.255, 10.1.2.0, 10.1.2.1 ... 10.1.2.255
10.1.1/28     -> 10.1.1.0 ... 10.1.1.255
10.1.1.0/28   -> 10.1.1.0 ... 10.1.1.15
10.1.1.0/30   -> 10.1.1.0, 10.1.1.1, 10.1.1.2, 10.1.1.3
10.1.1.128/25 -> 10.1.1.128 ... 10.1.1.255

Topic Format

TO subscribe to the pingmq results, you can use the following formats:

  • /ping/# will subscribe to both success and failed pings for all IP addresses
  • ping/success/+ will subscribe to success pings for all IP addresses
  • ping/failure/+ will subscribe to failed pings for all IP addresses
  • ping/+/8.8.8.8 will subscribe to both success and failed pings for all IP 8.8.8.8

Building

To build pingmq, you need to have installed Go 1.3.3 or 1.4. Then run the following:

# go get github.com/surge/surgemq
# cd surgemq/examples/pingmq
# go build

After that, you should see the pingmq command in the pingmq directory.

Archive