Introduction

Follow this guide to get up and running with CentOS 7, InfluxDB and Grafana. Once you complete this setup, you'll be collecting and displaying realtime performance metrics from your favorite API.

Note: This is a long read, at ~2300 words. It should take about an hour to get through the deployment. If you need some motivation get coffee'd up and review the example charts at the end of the post.

This post is fully updated to reflect the latest versions. Also, this article was moved from our previous hosting over to ghost. Enjoy!


Objective

We cover the topics below. No prior knowledge is required.

  • Download CentOS 7 Linux ISO
  • Create a virtual machine
  • Update CentOS 7
  • Configure CentOS 7 (CLI or GUI Desktop)
  • Install InfluxDB (time-series database)
  • Create two InfluxDB databases (iops and compute)
  • Install Grafana (web interface to display our metrics)
  • Point Grafana to an InfluxDB datasource
  • Suggest techniques to gather datapoints

Note: This can also be deployed on Windows and many other operating systems. See the official InfluxData and Grafana websites for downloads and instructions.


About Collecting Stats

After completing the setup, you can populate datapoints using your own scripts, or something like my vFlux-Stats-Kit on github.

By following this guide exactly, you will have the two databases that the above scripts expect. However, you are free to name the databases as you choose and fill them with any datapoints desired.

System Requirements

These packages run great on the smallest imaginable virtual machine. However, the proper setup should be a minimum of the following:

1 Virtual Machine (minimum)
CentOS 7 (Minimal Install is fine)
2 CPU (4 CPU recommended)
8GB RAM
40GB Harddrive* (80GB+ recommended)

Tip: If you need to resize your disk, do so before powering on. For existing virtual machines consult http://kb.vmware.com/kb/1006371

In this exercise, we use a single disk for simplicity. In a high performance deployment, you may want dedicated disks for InfluxDB.

Install CentOS 7

Skip this if you already have the OS up and running.

Step 1. Download

Step 2. Create

  • Create a virtual machine (i.e. on VMware Fusion or VMware vSphere)
  • Point the virtual machine to the ISO
  • Select the option to edit virtual hardware before powering on
  • Remove any extra devices (i.e. Camera, Floppy)
  • Size the disk
  • Configure CPU count
  • Configure Memory
  • Optionally, set the BIOS to UEFI

Note: CentOS 7 supports both Legacy BIOS or UEFI

Step 3. Power

  • Power On virtual machine
  • The CentOS splash screen appears
  • Select the option to install to disk
  • Choose the guided wizard instead of text only
  • Configure network, partitioning, NTP, etc.
  • Begin the Install

Note: We are not done yet! While the install runs perform the next step.

Step 4. Users and Passwords
While the install is running, we can perform additional tasks such as setting the root password and creating a user. Click the appropriate icon to perform each of the following:

  • Set root password
  • Create a user

Tip: Throughout this article, we begin as root for the initial updates, etc. We then start using sudo when adding additional components. Please adjust as needed.


Configure CentOS (CLI or GUI)

We can run in CLI-only mode, or add a desktop environment. You can also start with CLI mode and then add a desktop later. Choose an option below.

Option A - CLI Setup
If you will use CLI only (i.e. no GUI), then follow these steps.

  1. Login as root
  2. Configure an IP address using the nmtui text-based user interface tool
  3. Perform a systemctl restart network
  4. Test your success so far ping google.com
  5. Install VMware Tools with yum install open-vm-tools
  6. Update the OS and existing packages with yum update
  7. [optional] To allow root login via SSH, use the vi editor to uncomment the root login restriction.
    We will do this with vi /etc/ssh/sshd_config
  8. [optional] If you made the change in #7 above, perform a systemctl restart sshd.service
  9. If desired, confirm SSH access (i.e. with putty)

Option B - Desktop Setup
To deploy a desktop environment on CentOS 7, follow the steps below.

Here, we start with the CentOS 7 Minimal download. We then update, and add a desktop environment. We expect you to have CentOS 7 installed already, and be logged in as root (or use sudo).

Let's start by updating and adding some packages we need.

yum -y update
yum -y install epel-release
yum -y install open-vm-tools-desktop

Observe the above; When configuring CentOS 7 for desktop mode, we use open-vm-tools-desktop instead of open-vm-tools like we did for CLI mode earlier.

Next, we will install the KDE desktop environment.

yum -y groups install "KDE Plasma Workspaces"

Note: Replace the portion in quotes with "GNOME Desktop" if you prefer that.


Show Startup (CLI or GUI)
Determine if we are currently configured for CLI or GUI.

systemctl get-default

Configure Startup Environment
Set the default environment to be GUI.

set-default graphical.target

Reboot CentOS
From a terminal enter the following, or reboot however you like.

sync;sync;init 6

Login to Desktop
After the CentOS system reboots, login to the desktop as the user you created during the install. If you did not create a user yet, do that now before proceeding.

From here on, we issue commands with sudo.


Installing InfluxDB on CentOS 7

To install InfluxDB on CentOS, there are a few options. For this example, we will create a yum repository and install from there.


Add InfluxDB to yum
Just paste this entire snippet into your putty or terminal session:

cat <<EOF | sudo tee /etc/yum.repos.d/influxdb.repo
[influxdb]
name = InfluxDB Repository - RHEL \$releasever
baseurl = https://repos.influxdata.com/rhel/\$releasever/\$basearch/stable
enabled = 1
gpgcheck = 1
gpgkey = https://repos.influxdata.com/influxdb.key
EOF

Install InfluxDB

sudo yum install influxdb

We now have several binaries added to /usr/bin.

influx
influxd
influx_inspect
influx_stress
influx_tsm

influx is the CLI tool to manage things.
influxd is the actual binary that runs the program itself.


Start InfluxDB

sudo systemctl start influxdb

The influxd Process
Let's observe the influxd process and learn which configuration file it is pointing to:

ps -waux | grep influxd

You should see something like:

/usr/bin/influxd -config /etc/influxdb/influxdb.conf

View the contents:

cat /etc/influxdb/influxdb.conf

Or the more official way:

influxd config

for more detail on using configuration files with InfluxDB, see https://docs.influxdata.com/influxdb/v1.5/administration/config/


Allow InfluxDB through Firewall

sudo firewall-cmd --permanent --zone=public --add-port=8086/tcp
sudo firewall-cmd --permanent --zone=public --add-port=8083/tcp
sudo firewall-cmd --reload

Configure InfluxDB for Automatic start-up

sudo systemctl enable influxdb.service

Using the influx CLI
To interact with your installation of InfluxDB (i.e. create users, databases, etc.) perform the following:

  1. Launch terminal from within CentOS or SSH
  2. Change directory to /usr/bin
  3. Type influx and hit enter
mike@centos bin]$ influx
Connected to http://localhost:8086 version 1.5.2
InfluxDB shell version: 1.5.2
> 

Create InfluxDB Users
For this example, create a user called esx with a password of esx. Type the following into your influx CLI session and press enter.

CREATE USER esx WITH PASSWORD 'esx' WITH ALL PRIVILEGES

Tip: Influx commands only return interactive messages on failure. So after hitting enter above, if you get no feedback, this is good.


Note: For full details on InfluxDB authentication:
https://docs.influxdata.com/influxdb/v1.5/query_language/authentication_and_authorization/


Create InfluxDB Database
For this example, create a database called iops and another called compute.

CREATE DATABASE iops
CREATE DATABASE compute

Show InfluxDB Databases
From theinflux CLI, we can use the SHOW DATABASES command. We will always see _internal which is the default used by the system. We can also see the databases we added (iops and compute).

PS /home/mike> influx

Connected to http://localhost:8086 version 1.5.2
InfluxDB shell version: 1.5.2
> SHOW DATABASES
name: databases
name
----
_internal
iops
compute
> 

Note: We will use these databases in the next steps.


Install Grafana

First we will install dependencies. Since you probably already have these, you will likely see a message saying nothing to do; this is fine.

sudo yum -y install initscripts fontconfig

Additional font support (recommended)
Add the following packages to allow server-side image rendering.

sudo yum -y install fontconfig freetype* urw-fonts

Next, use touch to create a file called grafana.repo:

sudo touch /etc/yum.repos.d/grafana.repo

Edit the file with nano or vi. We will use vi:

sudo vi /etc/yum.repos.d/grafana.repo

Next, press i to enter insert mode and then paste in the following contents:

[grafana]
name=grafana
baseurl=https://packagecloud.io/grafana/stable/el/7/$basearch
repo_gpgcheck=1
enabled=1
gpgcheck=1
gpgkey=https://packagecloud.io/gpg.key https://grafanarel.s3.amazonaws.com/RPM-GPG-KEY-grafana
sslverify=1
sslcacert=/etc/pki/tls/certs/ca-bundle.crt

To save and exit in vi, press <ESC> on your keyboard and then type:

:wq!

Press <Enter> on your keyboard to complete the command.

Install Grafana

sudo yum -y install grafana

About Grafana Configuration
If desired, you can modify the Grafana configuration file at /etc/grafana/grafana.ini.


Set Grafana for automatic start-up

sudo systemctl enable grafana-server.service

Start Grafana Now

sudo /bin/systemctl daemon-reload
sudo /bin/systemctl enable grafana-server.service

Allow Grafana through firewall
By default, the grafana port is 3000. Adjust as needed.

sudo firewall-cmd --permanent --zone=public --add-port=3000/tcp
sudo firewall-cmd --reload

Check Grafana Service Status

sudo systemctl status grafana-server

Start Grafana
You may need to start, stop, or restart the service, especially after making changes like we did. The following will start the service:

sudo systemctl start grafana-server

Checking your work
Confirm that both InfluxDB and Grafana are configured for auto-start.

sudo systemctl list-unit-files | grep 'influx\|grafana'

The output from above should look similar to:

grafana-server.service                      enabled
influxd.service                             enabled
influxdb.service                            enabled

The Grafana Web Interface
The default login is admin admin. Here, you can point Grafana to one or more InfluxDB databases. Later, we can create graphs of performance data through this interface as well.

http://<ip address>:3000 

Keep this web page open for the next step.


Grafana Datasource

Grafana can visualize datapoints from many sources. You can add one or more datasources to a graph. Here we will add one, and make it the default.

In the beginning of this post, we created a database with InfluxDB. In this step, we point Grafana to the database as a datasource. The database is currently empty, but that does not matter.

In the web interface (i.e. http://127.0.0.1:3000) navigate to the Gear icon, and click Datasources. Click add, and then point to what will be our Default database.

Each time we create a new graph or table in Grafana, this datasource will automatically be selected. We can then add more datasources, or change it by clicking Edit on the Chart and changing the datasource.

Common Error
If you get a Network Error: Bad Gateway(502) when testing your datasource, scroll up to the HTTP section and click on your address; it will change from gray to white indicating it has been selected. Save and retest and you should be good to go.

Next Steps
Now you can start loading up the database(s) with performance stats from your VMware environment. For this task you can use your own scripts, or my vFlux Stats kit.

About vFlux Stats (optional)
Optionally, download my vFlux Stats Kit from github. These are PowerCLI scripts that connect to VMware vCenter Server and return performance data for virtual machines and ESXi hosts. The data is then written to your InfluxDB database, and served up by Grafana. Currently requires the scripts to be run from a Windows device with PowerShell 3.0 or later (PowerShell 5.1 preferred) and PowerCLI 6.5.1 or later (PowerCLI 10.x preferred).

Example Graph

We can get details for the ESXi host, all virtual machines and even break out a single virtual machine. This example shows a single virtual machine for the past 6 hours.
Example-Chart---Single-Virtual-Machine

Q: In the above, can you find an argument to change the layout of the virtual machine (i.e. CPU, Memory, etc.)?

A. Perhaps we can argue that:

  • The core count should be decreased (i.e. from 4 vCPU to 2 vCPU)
  • Memory can be decreased (unless locked by an app such as SQL)
  • When the workload ramps up it primarily uses only 1 vCPU (or does it?)
  • If we look closely we can see new threads freely starting on the downswing of other threads that are tearing-down
  • We may need need to review the importance of snappy response (decrease core count) vs. the power of long running or multithreaded workloads (stay at 4 vCPU).
  • Beyond this, troubleshooting may be subjective and environmental :p

Other Datapoints

The world is a big place with many things to study and visualize. Perhaps you are analyzing the impact of solar storms and want those datapoints and not just boring servers! Maybe you want an overlay of the two! This is all open source and the creative freedom is yours.

Summary

In this write-up we installed and configured InfluxDB, Grafana and CentOS 7. We created databases on InfluxDB and connected it all together with the beautiful Grafana web interface. The next step is yours, populating datapoints (and bragging)!

Supporting Documentation
https://docs.influxdata.com/influxdb/v1.5/introduction/installation/
http://docs.grafana.org/installation/rpm/
https://github.com/vmkdaily/vFlux-Stats-Kit

Who to Follow
@InfluxDB
@grafana
@vmkdaily

APPENDIX

Additional supporting information and examples.


Example - Adding Datasource
Let's look at a datasource being added. In the terminal, we can see that I have started the influx cli, and issued the SHOW DATABASES command. In the browser, we can see a working example pointing to the InfluxDB database iops:
Example-Grafana-Datasource-Setup-iops

About Error 502
Now let's look at a common issue caused by the URL not being active. This is located in the HTTP section of add Datasources (scroll up).

The URL is the InfluxDB address and port. The trick is to click it to activate. If you do not, and try to test the datasource, you will get Network Error: Bad Gateway(502).

Even though you see your desired InfluxDB address (i.e. http://localhost:8086, if the text is gray it requires clicking (it's a dropdown box)!

URL Inactive:
Grafana-Add-Datasources-URL-Inactive

URL Active:
Grafana-Add-Datasources-URL-Active

Once you have activated the URL, click Save and Test. Good to go! Repeat for any other datasources.

A look at the finished datasources, with one selected as default:
Grafana-with-Two-InfluxDB-Datasources-

-end-