Search VMTN with REST API

LucD — Sun, 31 Dec 2017 18:53:19 +0000

REST API are (nearly) everywhere! VMware’s VMTN website is no exception. I already did a post on Automate Your VMTN Search, but that was entirely based on constructing URI and interpreting the returned webpages. For the occasion of the PowerCLI’s 10th Birthday session at VMworld, I wanted to produce some InfoGraphs on the PowerCLI Community. For those InfoGraphs I needed to harvest data from said VMTN Community, and I looked for a better way to do this. That is where the REST API offered by the Jive software, om which the VMTN website is hosted, came in handy.

The functions I ended up with, are also a good example of how easy it is to consume REST API through PowerShell. And they also show how the basic techniques to work with REST API can be reused. Check out my VMTNRest repo.

VMTN and Jive

The VMTN website is currently build on the Jive 8, and besides the difference in some names, the major components of a Jive 8 hosted platform can be found back. These components are quite simple, but one needs to understand what is where, to be able to work with the REST API.

On a side note: working with REST API is quite straight-forward, the difficult part consists of understanding the setup and the logic of the platform behind the REST API. And the Jive platform is no exception.

There are four major endpoint services in a Jive environment. Since, for now, we only want to query the VMTN Communities, we will primarily be calling the REST API against the Content and People service endpoints. This will allow us to work with all the threads and documents published in the VMTN communities. And it will also allow us to gather information about users and groups in the VMTN communities.

For a complete overview of the Jive REST API see the Jive v3 REST API Reference Documentation.

Jive REST API calls

Basically the Jive REST API calls are similar to most of the other REST API calls. You compose the URI, eventually with a number of optional keywords and specify the method to be used. The data is returned in JSON format.

But the like I mentioned before, there are a couple of Jive specific elements one needs to be aware off.

Let’s look at an example. The following is a typical thread as can be found in any of the VMTN communities. The numbers in these pictures refer to the Annotations further on in the post.

The content of the thread is again nothing out of the extra ordinary.

If we “Get” that same thread through the REST API, there are a number of things to take note of.

By the way, this last picture is the output of the REST API call made through the Chrome RestLet Client, but could have been done through any other REST client, like for example PostMan.

Annotation

Like any other REST API call, each call is composed of three components
1. The VMTN Jive community location, i.e. https://communities.vmware.com
2. The Jive REST API prefix, i.e. /api/core/v3
3. The REST endpoint, i.e. contents
4. Optional fields. In this example we ask the endpoint to return just one result at a time, and to only return entries of type ‘discussion’
The Jive REST API accepts three types of authentication. In our example we used Basic Authentication. The authentication string is passed as a Header in the REST API call. We also specify that all data shall be in the JSON format. Update: it looks as if Basic Authentication to call the REST API is not supported in the current Jive version for “federated” user accounts. I’m still looking into that.
The “throw” line is is a security measure. It avoids high-jacking a Jive session through a so-called XSS attack in some browsers. Since we are not accessing the REST endpoint through a browser, we can just ignore the “throw” line.
The links field contains the URI for the previous and next page. If an endpoint returns more than a page-full of objects, the “next” field can be used to go to the next set of objects. Similarly, the URI in the “previous” field can be used to go back in the data.
Under the “list” field, the actual data concerning a Jive resource is returned.
Each Jive object returned by the REST endpoints, contains a “resources” field. In the example we are looking at a “discussion” object. Under the resources field we find several links to related information. For example, the “attachments” field will show what actions (“POST” and “GET”) are allowed, and also the URI to access these attachments.
The returned discussion object has several fields that provide extra information. The object contains fields to indicate how many users are following the discussion, how many like a specific reply, when the discussion was created and last updated.
The “author” field provides information about the user that created the discussion. Again, the “resources” field provides links to access related information about the author.
The “content” of the discussion. Note that this is provided as HTML
Two important fields related to discussions are the “question” and “resolved” fields. The first one indicates if the discussion is actually marked as a question, the second one indicates if there has been an accepted answer to the discussion or not.
The “startIndex” field shows the index of the next reply in the discussion. This also allows us to determine how many answer there have already been to the discussion, just note that the index is zero-based
The “parent” for a discussion object is the place, or community in VMTN terminology, where the discussion was started.

The Script

The script is in fact a module, named VMTNRest. The module contains a number of functions to access VMTN communities, authors and discussions.

You always start off with the Initialize-VMTNRest cmdlet. This cmdlet sets some module-wide values, and originally it was the intention to allow the caller to pass credentials. But apparently the current Jive version doen’t allow REST API calls from “federated” users with Basic Authentication. See here and here for more information on that issue. I’m still looking for a possible bypass, but for now, the cmdlets in the VMTNRest module only allow access to the “public” VMTN resources. If you have access to private communities, I’m afraid that for now you will not be able to access these resources.

Another parameter that could be of use is the MaxCount parameter. It defines how many objects the REST API returns on one call. The default is 100, also the maximum the Jive software allows. You can set this to a number between 1 and 100.

The MaxSamples parameter, which is available on the other cmdlets in the VMTNRest module, defines the maximum total number of objects that are returned in one call to the cmdlet. This parameter is set by default to 100, but can be changed. Be warned though, there are a lot of objects available in some categories. You don’t want to wait for example till the REST API has returned all +3400000 author objects

The other three public cmdlets retrieve objects from the VMTN community. They are Get-VMTNAuthor, Get-VMTNCommunity and Get-VMTNContent. Some of these objects are quite rich in content, as we will see in the next section.

Use Cases

As we said earlier, you always have to start with the Initialize-VMTNRest cmdlet. In the following example we will take all the defaults, and not use a proxy.

One way to get to know the VMTN platform, is to have a look which communities and groups are available. The following example uses one keywword, and yes, I might be a bit biased in my choice of examples

Initialize-VMTNRest

Get-VMTNCommunity -Community PowerCLI |
Select-Object -Property name,description,type

This returns the following.

Note that “place” is the internal Jive name for what we know as a VMTN community. Also note that the keywork “PowerCLI” doesn’t seem to appear in the name of a returned community. The reason is that internally the REST API looks at more than just the name of a community.

If you want to look for an exact match in the name, you can use the ExactMatch parameter.

Initialize-VMTNRest

Get-VMTNCommunity -Community 'VMware PowerCLI' -ExactMatch  |
Select-Object -Property name,description,type

Now we only get the specific community we were looking for.

Besides communities, there are also blogs and groups available in VMTN.

Initialize-VMTNRest

Get-VMTNCommunity -Community PowerCLI -IncludeBlog -IncludeGroup |
Select-Object -Property name,type,description

And we get entries that have “PowerCLI” in their name or description.

All the documents and entries in communities and blogs have an author. With the Get-VMTNAuthor we can explore those.

Initialize-VMTNRest

Get-VMTNAuthor -Author lucd | 
Select displayName,id,
    @{N='Published';E={([DateTime]$_.published).ToLocalTime()}},
    @{N='Updated';E={([DateTime]$_.updated).ToLocalTime()}}

This returns

The REST API looked for all authors whose name starts with “lucd”. But we also can go for an exact match.

Initialize-VMTNRest

Get-VMTNAuthor -Author lucd -ExactMatch | 
Select displayName,id,
    @{N='Published';E={([DateTime]$_.published).ToLocalTime()}},
    @{N='Updated';E={([DateTime]$_.updated).ToLocalTime()}}

And we get the single object back.

We can also do some historic research with the DateTime parameters. Do we still have authors around that were created in the early days of VMTN. And we assume that the official start date was July 10th in 2003.

Initialize-VMTNRest

$origin = Get-Date "10 Jul 2003"
Get-VMTNAuthor -CreatedAfter $origin -CreatedBefore $origin.AddDays(1) | 
Sort-Object -Property {[int]$_.id} |
Select-Object -First 5 -Property displayName,id,
        @{N='Points';E={$_.jive.level.points}},
        @{N='Published';E={([DateTime]$_.published).ToLocalTime()}} |
Format-Table -AutoSize

Yup, still some of those first-day authors around.

Just wonder what happened with ids 1 to 7

Finally we get to the core of the VMTN communities, the discussions. Again, quite simple to query.

Initialize-VMTNRest

Get-VMTNContent -Keyword 'New-OSCustomizationSpec' |
select subject,status,resolved,replycount,viewcount,
    @{N='Author';E={$_.author.displayName}},
    @{N='Location';E={$_.parentPlace.name}},
    @{N='Published';E={([DateTime]$_.published).ToLocalTime()}},
    @{N='LastActivity';E={([DateTime]$_.lastActivityDate).ToLocalTime()}} |
Format-Table -AutoSize

And this returns 100 objects, remember, the default MaxSamples.

These “content” objects are very rich, but I leave that to you and Get-Member to discover.

As we stated earlier, with the cmdlet we can also search the blog posts and documents. Let’s check if someone blogged about VSAN in the last month.

Initialize-VMTNRest

$now = Get-Date

Get-VMTNContent -Keyword 'VSAN' -ContentType post -SearchSubjectOnly -Start $now.AddMonths(-1) | 
select subject,status,resolved,replycount,viewcount,
    @{N='Author';E={$_.author.displayName}},
    @{N='Location';E={$_.parentPlace.name}},
    @{N='Published';E={([DateTime]$_.published).ToLocalTime()}},
    @{N='LastActivity';E={([DateTime]$_.lastActivityDate).ToLocalTime()}} |
Format-Table -AutoSize

And there we go.

The functionality these cmdlets bring you is of course also available via a Web Browser and the Search functionality on the VMTN website. But with these cmdlets you can now automate some of your searches. You could for example schedule a script to check weekly if something was published about VSAN. Mail the report to yourself, and you can immediately see if you need to fire up your browser.

The cmdlets are just another way to access the information available in VMware’s VMTN Community, and seen the richness of the returned objects, you can go very far in this. Some examples are the InfoGraphs I ended up with in the end for my VMworld session.

and

If you have suggestions to improve or expand the VMTNRest module, feel free to contact me.

Enjoy!

Automate your VMTN search

LucD — Wed, 29 Feb 2012 21:26:54 +0000

Recently I had the pleasure of doing a guest post, called Finding your way in the PowerCLI Community, on the PowerCLI blog. The subject of the post was how to find community threads, that might hold an answer to your question.

Now this wouldn’t be a PowerShell/PowerCLI blog, if I didn’t try to automate the procedure. And with a serious amount of RegEx involved, I was able to create some working code. Here it is, my Find-VMTNPowerCLI function.

Warning: pure PowerShell, no PowerCLI content !

The script

function Find-VMTNPowerCLI{
<#
.SYNOPSIS  Search the VMTN PowerCLI related communities
.DESCRIPTION The function automates the functionality that
  is available through the Search page
.NOTES  Author:  Luc Dekens
.PARAMETER Question
  The question as you would enter it on the VMTN Search page
.PARAMETER Start
  The start date of the period in which you are looking for
  threads that fitt your search.
  The default is December 24th 2007.
.PARAMETER Finish
  The start date of the period in which you are looking for
  threads that fitt your search.
  The default is now.
.PARAMETER ReplyStart
  The start date of the period in which you are looking for
  replies that fitt your search.
  The default is December 24th 2007.
.PARAMETER ReplyFinish
  The start date of the period in which you are looking for
  replies that fitt your search.
  The default is now.
.PARAMETER Author
  Filter the returned threads to those started by one of the
  authors.
.PARAMETER ReplyAuthor
  Filter the returned threads to those where the reply came
  from one of the authors.
.PARAMETER PowerCLI
  Switch, search the PowerCLI Community.
  The default for this switch is $true.
.PARAMETER Onyx
  Switch, search the Onyx Community.
  The default for this switch is $false.
.PARAMETER UpdateManager
  Switch, search the UpdateManager Community.
  The default for this switch is $false.
.PARAMETER PowerShellers
  Switch, search the PowerShellers group.
  The default for this switch is $false.
.PARAMETER AutomationTools
  Switch, search the AutomationTools COmmunity.
  The default for this switch is $false.
.PARAMETER Descending
  Switch, when true the returned threads will be
  in a descending Reply Date order.
  The default for this switch is $true.
.EXAMPLE
  PS> Find-VMTNPowerCLI -Question 'svmotion thin'
.EXAMPLE
  PS> Find-VMTNPowerCLI -Question '5.0.1' -PowerShellers
.EXAMPLE
  PS> Find-VMTNPowerCLI -Question 'regex get-view' `
  >> -Start (Get-Date).AddDays(-14)
#>

  param(
  [CmdletBinding()]
  [parameter(Mandatory = $true)]
  [string]$Question,
  [datetime]$Start = (New-Object DateTime -ArgumentList 2007,12,24),
  [datetime]$Finish = (Get-Date),
  [datetime]$ReplyStart = $Start,
  [datetime]$ReplyFinish = $Finish,
  [string[]]$Author,
  [string[]]$ReplyAuthor,
  [switch]$PowerCLI = $true,
  [switch]$Onyx,
  [switch]$UpdateManager,
  [switch]$PowerShellers,
  [switch]$AutomationTools,
  [switch]$Descending = $true
  )

  function Get-VMTNSearch{
    param(
    [string]$url,
    [switch]$Group
    )

    $pst = [TimeZoneInfo]::FindSystemTimeZoneById("Pacific Standard Time")
    $t0 = Get-Date
    $t1 = [TimeZoneInfo]::ConvertTime($t0,$pst)
    $tdiff = $t0 - $t1

    $regBodyText = [regex]'(?s)\[DocumentBodyStart.*?-->(?.*)

VMTN Archives - LucD notes