Search VMTN with REST API

REST API are (nearly) everywhere! VMware’s VMTN website is no exception. I already did a post on Automate Your VMTN Search, but that was entirely based on constructing URI and interpreting the returned webpages. For the occasion of the PowerCLI’s 10th Birthday session at VMworld, I wanted to produce some InfoGraphs on the PowerCLI Community. For those InfoGraphs I needed to harvest data from said VMTN Community, and I looked for a better way to do this. That is where the REST API offered by the Jive software, om which the VMTN website is hosted, came in handy.

The functions I ended up with, are also a good example of how easy it is to consume REST API through PowerShell. And they also show how the basic techniques to work with REST API can be reused. Check out my VMTNRest repo.

VMTN and Jive

The VMTN website is currently build on the Jive 8, and besides the difference in some names, the major components of a Jive 8 hosted platform can be found back. These components are quite simple, but one needs to understand what is where, to be able to work with the REST API.

On a side note: working with REST API is quite straight-forward, the difficult part consists of understanding the setup and the logic of the platform behind the REST API. And the Jive platform is no exception.

There are four major endpoint services in a Jive environment. Since, for now, we only want to query the VMTN Communities, we will primarily be calling the REST API against the Content and People service endpoints. This will allow us to work with all the threads and documents published in the VMTN communities. And it will also allow us to gather information about users and groups in the VMTN communities.

For a complete overview of the Jive REST API see the Jive v3 REST API Reference Documentation.

Jive REST API calls

Basically the Jive REST API calls are similar to most of the other REST API calls. You compose the URI, eventually with a number of optional keywords and specify the method to be used. The data is returned in JSON format.

But the like I mentioned before, there are a couple of Jive specific elements one needs to be aware off.

Let’s look at an example. The following is a typical thread as can be found in any of the VMTN communities. The numbers in these pictures refer to the Annotations further on in the post.

The content of the thread is again nothing out of the extra ordinary.

 

 

If we “Get” that same thread through the REST API, there are a number of things to take note of.

 

By the way, this last picture is the output of the REST API call made through the Chrome RestLet Client, but could have been done through any other REST client, like for example PostMan.

Annotation

  1. Like any other REST API call, each call is composed of three components
    1. The VMTN Jive community location, i.e. https://communities.vmware.com
    2. The Jive REST API prefix, i.e. /api/core/v3
    3. The REST endpoint, i.e. contents
    4. Optional fields. In this example we ask the endpoint to return just one result at a time, and to only return entries of type ‘discussion’
  2. The Jive REST API accepts three types of authentication. In our example we used Basic Authentication. The authentication string is passed as a Header in the REST API call. We also specify that all data shall be in the JSON format. Update: it looks as if Basic Authentication to call the REST API is not supported in the current Jive version for “federated” user accounts. I’m still looking into that.
  3. The “throw” line is is a security measure. It avoids high-jacking a Jive session through a so-called XSS attack in some browsers. Since we are not accessing the REST endpoint through a browser, we can just ignore the “throw” line.
  4. The links field contains the URI for the previous and next page. If an endpoint returns more than a page-full of objects, the “next” field can be used to go to the next set of objects. Similarly, the URI in the “previous” field can be used to go back in the data.
  5. Under the “list” field, the actual data concerning a Jive resource is returned.
  6. Each Jive object returned by the REST endpoints, contains a “resources” field. In the example we are looking at a “discussion” object. Under the resources field we find several links to related information. For example, the “attachments” field will show what actions (“POST” and “GET”) are allowed, and also the URI to access these attachments.
  7. The returned discussion object has several fields that provide extra information. The object contains fields to indicate how many users are following the discussion, how many like a specific reply, when the discussion was created and last updated.
  8. The “author” field provides information about the user that created the discussion. Again, the “resources” field provides links to access related information about the author.
  9. The “content” of the discussion. Note that this is provided as HTML
  10. Two important fields related to discussions are the “question” and “resolved” fields. The first one indicates if the discussion is actually marked as a question, the second one indicates if there has been an accepted answer to the discussion or not.
  11. The “startIndex” field shows the index of the next reply in the discussion. This also allows us to determine how many answer there have already been to the discussion, just note that the index is zero-based
  12. The “parent” for a discussion object is the place, or community in VMTN terminology, where the discussion was started.

The Script

The script is in fact a module, named VMTNRest. The module contains a number of functions to access VMTN communities, authors and discussions.

You always start off with the Initialize-VMTNRest cmdlet. This cmdlet sets some module-wide values, and originally it was the intention to allow the caller to pass credentials. But apparently the current Jive version doen’t allow REST API calls from “federated” users with Basic Authentication. See here and here for more information on that issue. I’m still looking for a possible bypass, but for now, the cmdlets in the VMTNRest module only allow access to the “public” VMTN resources. If you have access to private communities, I’m afraid that for now you will not be able to access these resources.

Another parameter that could be of use is the MaxCount parameter. It defines how many objects the REST API returns on one call. The default is 100, also the maximum the Jive software allows. You can set this to a number between 1 and 100.

The MaxSamples parameter, which is available on the other cmdlets in the VMTNRest module, defines the maximum total number of objects that are returned in one call to the cmdlet. This parameter is set by default to 100, but can be changed. Be warned though, there are a lot of objects available in some categories. You don’t want to wait for example till the REST API has returned all +3400000 author objects 🙂

The other three public cmdlets retrieve objects from the VMTN community. They are Get-VMTNAuthor, Get-VMTNCommunity and Get-VMTNContent. Some of these objects are quite rich  in content, as we will see in the next section.

Use Cases

As we said earlier, you always have to start with the Initialize-VMTNRest cmdlet. In the following example we will take all the defaults, and not use a proxy.

One way to get to know the VMTN platform, is to have a look which communities and groups are available. The following example uses one keywword, and yes, I might be a bit biased in my choice of examples 🙂

This returns the following.

Note that “place” is the internal Jive name for what we know as a VMTN community. Also note that the keywork “PowerCLI” doesn’t seem to appear in the name of a returned community. The reason is that internally the REST API looks at more than just the name of a community.

If you want to look for an exact match in the name, you can use the ExactMatch parameter.

Now we only get the specific community we were looking for.

Besides communities, there are also blogs and groups available in VMTN.

And we get entries that have “PowerCLI” in their name or description.

All the documents and entries in communities and blogs have an author. With the Get-VMTNAuthor we can explore those.

This returns

The REST API looked for all authors whose name starts with “lucd”. But we also can go for an exact match.

And we get the single object back.

We can also do some historic research with the DateTime parameters. Do we still have authors around that were created in the early days of VMTN. And we assume that the official start date was July 10th in 2003.

Yup, still some of those first-day authors around.

Just wonder what happened with ids 1 to 7 😉

Finally we get to the core of the VMTN communities, the discussions. Again, quite simple to query.

And this returns 100 objects, remember, the default MaxSamples.

These “content” objects are very rich, but I leave that to you and Get-Member to discover.

As we stated earlier, with the cmdlet we can also search the blog posts and documents. Let’s check if someone blogged about VSAN in the last month.

And there we go.

The functionality these cmdlets bring you is of course also available via a Web Browser and the Search functionality on the VMTN website. But with these cmdlets you can now automate some of your searches. You could for example schedule a script to check weekly if something was published about VSAN. Mail the report to yourself, and you can immediately see if you need to fire up your browser.

The cmdlets are just another way to access the information available in VMware’s VMTN Community, and seen the richness of the returned objects, you can go very far in this. Some examples are the InfoGraphs I ended up with in the end for my VMworld session.

and

If you have suggestions to improve or expand the VMTNRest module, feel free to contact me.

Enjoy!

 

Leave a Reply

Your email address will not be published. Required fields are marked *

*
*

This site uses Akismet to reduce spam. Learn how your comment data is processed.