Using PowerShell to Get Data from the Web

Just a quick example to show how one can use PowerShell to access data on a website. In our case we will use a simple URL from vmware.com that serves up a useful file we may want to look at.

PowerShell Web Cmdlets

Think of this as the modern day replacement for cURL; Today we will focus on Invoke-WebRequest, a native PowerShell cmdlet available since PowerShell version 3.0. There is also a related cmdlet known as Invoke-RestMethod which is worth checking out.

Setup

We start with just the URI and saving the result to a variable.

$Uri = 'https://packages.vmware.com/tools/versions'
$result = Invoke-WebRequest -Uri $Uri

Show the result

Here we can see the variable now holds several properties. Notice that the Content property gives us a preview of the first 6 lines or so in this case.

PS C:\> $result


StatusCode        : 200
StatusDescription : OK
Content           : # VMware version-mapping file.
                    #
                    # This file provides a one-to-one mapping between...
                    # ESX/ESXi version-number codes, and paths to OSP...
                    # for that Tools version.
                    #
                    ...
RawContent        : HTTP/1.1 200 OK
                    Connection: keep-alive
                    Accept-Ranges: bytes
                    Content-Length: 8254
                    Content-Type: text/plain
                    Date: Tue, 24 Oct 2017 20:09:06 GMT
                    ETag: "1cbe5926328011f0454b2b5468839bd6:1506481047"...
Forms             : {}
Headers           : {[Connection, keep-alive], [Accept-Ranges, bytes],
Images            : {}
InputFields       : {}
Links             : {}
ParsedHtml        : mshtml.HTMLDocumentClass
RawContentLength  : 8254

Getting at the Content

PS C:\> $result.Content
# VMware version-mapping file.
#
# This file provides a one-to-one mapping between VMware Tools for
# ESX/ESXi version-number codes, and paths to OSP repositories suitable
# for that Tools version.
#
# The ESXi server mapping is only to show that the particular version of
# Tools ships with that particular ESXi server build number, but the Tools
# can work with a greater range of ESXi versions.
#
# Column 1: Tools version on NGC/VI Client
# Column 2: ESXi server version.'esx/0.0' indicates that the tools version
# is not yet bundled with ESXi.
# Column 3: Tools version on guest Setup/About page
# Column 4: ESXi server build number
#
10287       esx/0.0                                10.1.15
10282       esx/0.0                                10.1.10
10279       esx/6.5u1                              10.1.7        5969303
10277       esx/6.0p05                             10.1.5        5572656
10272       esx/6.5p01                             10.1.0        5146846
10272       esx/6.5                                10.1.0        4564106
10252       esx/6.5                                10.0.12       4564106
10249       esx/5.5p11                             10.0.9        6480324
10249       esx/5.5ep11                            10.0.9        5230635
10249       esx/6.0u3                              10.0.9        5050593
10249       esx/5.5p10                             10.0.9        4722766
10249       esx/6.0p04                             10.0.9        4600944
10249       esx/5.5p09                             10.0.9        4345813
10249       esx/6.0p03                             10.0.9        4192238
10249       esx/5.5p08                             10.0.9        4179633
10248       esx/0.0                                10.0.8
10246       esx/6.0u2                              10.0.6        3620759
10245       esx/0.0                                10.0.5
10240       esx/6.0ep05                            10.0.0        3566359
10240       esx/6.0p02                             10.0.0        3380124
10240       esx/5.5p07                             10.0.0        3248547
10240       esx/5.5ep10                            10.0.0        3568722
10240       esx/5.5ep09                            10.0.0        3343343
9541        esx/6.0ep04                            9.10.5        3247720
9541        esx/6.0u1                              9.10.5        3029758
9537        esx/6.0p01                             9.10.1        2809209
9536        esx/6.0                                9.10.0        2494585

Downloading a file

This one is easy, just don't don't forget the OutFile parameter or the file will not be downloaded. This example lets us download it to our local disk. This is great for downloading bits from vendors, etc. though in this case we are just looking at a published list.

Invoke-WebRequest -Uri $Uri -OutFile c:\temp\tools-mapping-file.txt

Do Something Useful

Since we have realtime access to this data, I don't want to download the file every time and look at it. I would rather take the data live and do something with it.

In my case, I wanted to add another column containing unique data points (versions impacted by a known issue with RSS). Also, I wanted to make it more consumable by returning things as objects so I could search for version numbers for example.

So what I did was parse the Content property we reviewed earlier, and then added each line to an array, so I could return objects. I also added switches that optionally maintain the original feel of the plain text document as served up on the web.

If you are interested, you can check out the module on github:
VMToolsOnline - PowerShell Module to list VMware Tools versions from the web

Enjoy!