Installing node_exporter on an EC2 Instance

I had a difficult time finding information on how to install node_exporter on an EC2 instance. The following will be rough, but hopefully educational…

Just as an FYI I am doing this on an EC2 instance running with amzn-ami-2018.03.f-amazon-ecs-optimized.

Install Script

My preferred method of running node_exporter is as a service. You can run the following in a bash shell or as a script:

#!/bin/bash
sudo useradd --shell /bin/false prometheus
sudo mkdir /var/log/prometheus/
sudo touch /var/log/prometheus/node_exporter.log
sudo chown -R prometheus:prometheus /var/log/prometheus
sudo cat > initscript <<'endmsg'
#!/bin/bash
#
# chkconfig: 2345 20 80 Read
# description: node_exporter will export system metrics in a Prometheus format
# processname: node_exporter

# Source function library.
. /etc/rc.d/init.d/functions

PROGNAME=node_exporter
PROGDIR=/home/prometheus/node_exporter
PROG=$PROGDIR/$PROGNAME
USER=prometheus
LOGFILE=/var/log/prometheus/$PROGNAME.log
LOCKFILE=/var/run/$PROGNAME.pid

start() {
echo -n "Starting $PROGNAME: "
cd $PROGDIR
daemon --user $USER --pidfile="$LOCKFILE" "$PROG &>$LOGFILE &"
echo $(pidofproc $PROGNAME) >$LOCKFILE
echo
}

stop() {
echo -n "Shutting down $PROGNAME: "
killproc $PROGNAME
rm -f $LOCKFILE
echo
}


case "$1" in
start)
start
;;
stop)
stop
;;
status)
status $PROGNAME
;;
restart)
stop
start
;;
reload)
echo "Sending SIGHUP to $PROGNAME"
kill -SIGHUP $(pidofproc $PROGNAME)
;;
*)
echo "Usage: service $PROGNAME {start|stop|status|reload|restart}"
exit 1
;;
esac
endmsg
sudo mv initscript /etc/init.d/node_exporter
sudo chmod +x /etc/init.d/node_exporter
sudo chown prometheus:prometheus /etc/init.d/node_exporter
curl -LO https://github.com/prometheus/node_exporter/releases/download/v0.16.0/node_exporter-0.16.0.linux-amd64.tar.gz
sudo mkdir /home/prometheus/node_exporter
sudo tar -xf node_exporter-*.tar.gz
sudo mv node_exporter-*/* /home/prometheus/node_exporter/
sudo chown -R prometheus:prometheus /home/prometheus/node_exporter
sudo chkconfig --add node_exporter
sudo service node_exporter start

 

The commands above will do the following

  1. Create a service user named prometheus
  2. Create a log file owned by the prometheus user to be used by the node_exporter service
  3. Create a script in the /etc/init.d directory to allow running the node_exporter executable as a service
  4. Download version 0.16.0 of node_exporter and untar it into the /home/prometheus/node_exporter directory
  5. Set the node_exporter service to automatically start on boot
  6. Start the node_exporter service

 

After the script has successfully executed, you can start/stop the service using the command service node_exporter start or service node_exporter stop. You can also check the status of the service using service node_exporter status.

Advertisement

Using cgo to call C code from within Go code

I lack a deep understanding of how C works, but I wanted to call some of the functions within the pdh.dll Windows library from my Go code. I explored a few rabbit holes, so I figured I would post how I got this working. If you want to skip to running my examples on your own, you can download them here.

First off, if you don’t know anything about cgo (like myself) then I recommend learning the basics by reading C? Go? Cgo!

Before running any code using cgo or gcc you have to have gcc installed on your machine. I found the easiest install method to be tdm-gcc. Simply download the file corresponding to the architecture of your machine then run it.

Including a Windows Library

In my example code, pdh.go demonstrates what I set out to do. Here is the code:

package pdh

// #cgo LDFLAGS: -L./lib -lpdh
// #include <pdh.h>
import "C"

func PdhValidatePath(path string) uint32 {
    ret := C.PdhValidatePath(C.CString(path))
    return uint32(ret)
}

// #cgo LDFLAGS: -L./lib -lpdh tells the gcc compiler to look for the pdh library within the standard lib directory (note that this directory is not within the Go project directory).

// #include <pdh.h> tells the gcc compiler to include pdh.h. This means that all of the functions within pdh.h can be called using C.<function name>. An example of this is shown by the line ret := C.PdhValidatePath(C.CString(path))

Including Your Own C

If you want to write your own C class, then I recommend taking a look at the package named increment within my example. In the package directory you can see increment.c and increment.h. These files were translated into increment.lib by running the following commands:

gcc -O2 -c increment.c
ar q increment.lib increment.o

Since increment.lib and increment.h exist within the package directory, they can be referenced with just a simple #include statement as shown here:

package increment

// #include <increment.h>
import "C"

func Increment(i int) int{
   return int(C.x(C.int(i)))
}

Including a Standard Library With Custom C Code

My package titled stdlib demonstrates how to include a standard library with custom C code that references that standard library.

package stdlib

/*
#include 
#include 

void myprint(char* s) {
	printf("%s\n", s);
}
*/
import "C"

import "unsafe"

func PrintText() {
	cs := C.CString("Hello from stdio\n")
	C.myprint(cs)
	C.free(unsafe.Pointer(cs))
}

func Random() int {
	return int(C.rand())
}

func Seed(i int) {
	C.srand(C.uint(i))
}

In this code, you can see that there is a C function called myprint that is defined in the cgo preample. The myprint function can be used to print text using the stdio printf function. This is a pretty simple example, but it is a good starting point for anyone that has a reason to write code in C rather than Go.

Go Glob Case Insensitive

With Go, the filepath Glob() function is a really fast way to find all files that match a particular path, with optional wild cards. I was disappointed, though, that there is not a way to force case-insensitivity to be performed by the Glob() function to the entire pattern that is passed in.

What I found was that the Glob() function is case-insensitive on Windows and case-sensitive on Linux, due to the way that each OS handles file paths. With a regex pattern you can easily force case-insensitivity to the entire pattern using (?i), unfortunately this doesn’t work with Glob() because it doesn’t use regex patterns. Don’t give up hope, though, there is a way to do this.

The Glob() function allows the use of the ‘[‘ and ‘]’ runes to define patterns within the pattern string that is passed into the Glob() function. So the best way I found to do a case-insensitive search is to replace every Letter category Unicode code-point (aka rune) with a lowercase and uppercase version of the rune surrounded by square brackets. So ‘a’ would become ‘[aA]’ and ‘test’ would become ‘[tT][eE][sS][tT]’. In case you don’t know what is included in the Unicode Letter category, you can view the contents here. I believe that the Letter category is made up of the combination of the Ll, Lm, Lo, Lt, and Lu categories. There is no need to alter the path on Windows, since it is already case-insensitive, so I wrote a function to only be performed if the OS is not Windows.

Here is the Go code that I used to perform a conversion of a file path string into a case-insensitive version of the same file path:

func InsensitiveFilepath(path string) string {
   if runtime.GOOS == "windows" {
      return path
   }

   p := ""
   for _, r := range path {
      if unicode.IsLetter(r) {
         p += fmt.Sprintf("[%c%c]", unicode.ToLower(r), unicode.ToUpper(r))
      } else {
         p += string(r)
      }
   }
   return p
}

Here is what it looks like when it runs:

fmt.Println(InsensitiveFilepath("/home/user/test/*"))

/[hH][oO][mM][eE]/[uU][sS][eE][rR]/[tT][eE][sS][tT]/*

You can play with it yourself on the the Go playground!

Go File Permissions on Linux

I wrote an application in Go, aka Golang, that I use to search for text matching a regex pattern within text files. Up to now, everything has worked well with my testing on Windows 10. However, I put my compiled application on to a Linux machine (specifically RHEL) today and I was getting errors with my os.MkdirAll() call. The function call looked like this in my code:

if _, err := os.Stat(logDirectory); os.IsNotExist(err) {
   os.MkdirAll(logDirectory, 0777)
}

Basically this code creates a directory using the path specified in the variable logDirectory with the permissions 777 (Read, Write, and Execute for all users) if the directory does not already exist. If the directory does not exist it will create the directory as well as any necessary parent directories that don’t already exist.

What I found is that my application creates the non-existent directory, but the permissions specified (0777) aren’t what are actually applied to the newly created directory. What ends up being applied is the default permissions according to the umask that is set. So you can see below that my umask is set to 0777 (meaning grant no permissions to newly created files and directories) so the newly created directory has the permissions 0000.

filepermissions.png

If you are unfamiliar with umask, like I was, then here is a basic description as I understand it. Umask is the default permission given when a new file or folder is created on a Linux/Unix machine. Umask accepts values between 0 and 7 that are the negated form of normal permissions. Meaning if you want the default permissions on a directory to be 765 then you should set the umask value to 012. Or if you want the default permissions to be 555, then you should set umask to 222. You can set umask in a shell simply by typing ‘umask’ followed by the desired value. You can view what umask is set to by entering ‘umask’ with no value after it. For example:

umask.png

So back to the issue my Go application was experiencing. I thought this must be some sort of bug when running Go on Linux. Through some Googling, I found that many other Go users have thought the same thing as me. However, the creators of Go state that this is not a bug. Basically their answer is that Go is built to run on various OS’s, but it still has to honor the behaviors expected on each OS. Meaning that in my case, I should not expect file permissions to be handled consistently between my testing on Windows 10 and Linux RHEL. This makes sense. Although, it makes me wonder why the Go MkdirAll() function accepts a parameter for setting the permissions if it isn’t going to be honored.

The Solution

The solution I was able to come up with to solve this issue is that you must use the os.Chmod() function to set the permissions of a directory after creating the directory with the os.MkdirAll() function. Yes it seems weird that I am stating in code to set the permissions to 0777 twice, but that’s the way it is. So here is my updated code:

if _, err := os.Stat(logDirectory); os.IsNotExist(err) {
   os.MkdirAll(logDirectory, 0777)
   os.Chmod(logDirectory, 0777)
}

Here are the results of my test on Linux using this new code:

filepermissionswithchmod.png

As you can see, umask was set to 0777 which means that a newly created directory should have the default permissions of 0000. However, thanks to the addition of os.Chmod() the logs directory now has the permissions 0777. Now that the logs directory has the desired permissions, my Go application is able to run as expected and create new log files when needed.

Additional Information

Unix Permissions Calculator

Umask Explanation

Discussion on how Go’s os.Mkdir() function should handle permission bits.

Linux Chmod Command

Password Management

In today’s world there is so much risk in not having safe passwords. Having safe passwords can be a bit of a hassle, though, when you have hundreds to remember. That is why you need a password management tool, so you don’t have to remember the passwords.

There are many different password manager apps that you can get these days. Some are free, some are not. I tried LastPass, Dashlane, 1Password, KeePass, and a few others. Ultimately I decided that KeePass was good enough, and it is free. If you know me, you know that I like free.

Getting Started

So how do you get started with KeePass? You will need to download a KeePass app on each of your devices that you want to use for viewing passwords. I also suggest using some sort of cloud drive system for synchronizing your database file between your devicse. I chose to use Google Drive simply because I already use Google Drive. You could use any others like One Drive, Dropbox, etc.

Here are links to each of the downloads I used:

  • Windows
  • Mac or Linux
    • KeePassXC
      • I chose this app because the original KeePass app is built for Windows and I found that it didn’t run very well on Mac with Mono. KeePassXC is a native Mac OS app and runs great. In fact it is also compatible with Windows.
    • Google Drive
  • Android
    • Keepass2Android
      • I chose this because it has a built in Android Keyboard that can be used to pull credentials out of my KeePass database without leaving the app where I am entering credentials.
    • Google Drive

Creating a Database File

After installing all of the necessary apps on my devices, I created my KeePass database file by doing the following:

  1. Open KeePass 2 on my Windows Laptop (you could do this step on any of your devices)
  2. Click the New icon at the top leftkeepass_new.png
  3. Select a location to store the database file and give it a name. I selected the Google Drive folder on my laptop as the destination for my database file.keepass_drive.png
  4. Enter a master password. This password will be the only password you will ever need to remember from now on, and it will be responsible for keeping all your other passwords safe. So chose a long secure password that you can easily remember. A good way to do this is to use a sentence or combination of easy to remember words.keepass_masterpassword.png
  5. Name your database. This step isn’t really important unless you’re planning to have multiple databases. I just gave it a simple name.keepass_databasename.png
  6. Add all your passwords one at a time by clicking the Add Entry button at the top left corner. At a minimum, you should give each entry a title (like gmail), a user name, and a password.keepass_addentry.png
  7. Click the save icon at the top left
  8. Now I have a database

For additional information you can view the KeePass help.

Synchronization

Since I placed my database file within my google drive folder, it will automatically be synced to Google Drive. Therefore, it is crucial that my google account be secure. I have 2 step verification enabled on my Google account. I highly suggest you do the same. If you don’t know how, follow these instructions.

I can now open my database file on my Macbook simply by opening the KeePassX application and then selecting the option to open an existing database file. On my Macbook the database file appeared in my Google Drive folder, so I simply selected that and clicked open. After clicking open, it asks for the master password that I entered when I created my database file before. Now that I have the database opened, I can add entries and click save, then the database gets synced back to Google Drive and to my other devices. I suggest only modifying the database on 1 device at a time. If you make modifications on 2 devices then click save then you will get a merge conflict and Google Drive will ask you to select which copy to keep.

On my Android device I opened Keepass2Android, selected Open Database, selected Google Drive, then clicked on my database file. I enter the master password, and boom! I now have access to all of my passwords.

Auto-Type

There is a handy option available in KeePass called Perform Auto-Type. What this does is it will automatically type in the username and password for the particular entry for you. To use this, go to the website or app where you want to login and place your cursor in the username field. Open KeePass and right click on the entry you want to use and click Perform Auto-Type (or type ctrl+v/cmd+v). KeePass will automatically switch back to your other app, type in the username, tab to the next field, type in the password, then hit enter. This works in most cases, but one problem I found is that when logging into some services you need to enter 3 entries. Like maybe a domain or account name. One example of this is logging into the AWS Console. In order to handle these cases you can modify the Auto-Type procedure for a particular entry. Here is how I made a custom sequence for the AWS Console:

aws.png

  1. Double click on the AWS credential entry in KeePass
  2. Click on the Auto-Type tab at the top
  3. Select the radio button that says Override default sequence
  4. Enter in your custom sequence. The AWS Console webpage loads with the cursor in the username field by default. To handle this, I used a custom sequence that first does a shift+tab to go up to the Account ID field, then it tabs down one at a time to the username and password fields. This sequence looks like this (note that the shift key ‘+’ doesn’t work in KeePassXC): +{TAB}{S:Account}{TAB}{USERNAME}{TAB}{PASSWORD}{ENTER}autotype.png
  5. Save the sequence. Don’t forget to save your database file after making changes to any database entries.
  6. Now when I load the AWS Console page all I have to do is click Perform Auto-Type and all 3 fields are populated for me.

 

autotypebutton.png

MFA or TOTP Keys

It is a good idea to use multi-factor authentication, or 2 step verification, when possible. If you are not familiar with this, basically it means that after you enter your username and password you are prompted for an additional password (or key) that is generated for you. A common way to get this generated key is by using the Google Authenticator app on your smart phone. This makes it much more difficult for people to get into your accounts. Another way to generate the MFA key, which I think is very convenient, is by using the KeePass app with an installed plugin. I have this working on my Windows laptop. If you use KeePassXC then there is no need to install the TOTP plugin because there is already one built in, so skip down to step 6 (the steps will be a little different, but similar in KeePassXC). Here are the steps I took on Windows to get this working:

  1. Close your KeePass application
  2. Download the latest version of Tray TOTP
  3. Open file explorer and browse to the directory where you have KeePass installed. This is usually C:\Program Files (x86)\KeePass Password Safe 2\
  4. Copy the downloaded file (TrayTotp.plgx) and paste it into the Plugins directory within your KeePass installation directorykeepass_totp.png
  5. Open KeePass and enter your master password to gain access to your database
  6. Now you can add a TOTP or MFA key to any of your saved credentials by right clicking on the entry, highlight Selected Entry, then click on Setup TOTPkeepass_totp_add.png
  7. Click next on the first few prompts until you get to the prompt that asks you to choose between 6 or 8 for the length of the key. This will depend on which service you are generating a key for. For Google accounts and AWS accounts I know that you can choose 6.keepass_totp_6.png
  8. Now you are prompted for your TOTP seed. This is generated by the service for which you are using MFA. As an example I will use a Google account. When turning on 2 Step Authentication you will click the Set Up button to use the Google Authenticator App.authenticator.png
  9. You can select Android, then you will be presented with a QR code to scan. If you want to have the ability to use the 2 Step Authentication with your phone, I suggest you scan the QR code using the Google Authenticator app now.
  10. To get the key working with KeePass you will need to click the button that says CAN’T SCAN IT?authenticator_cantscan.png
  11. Now you will see an alphanumeric key. Copy this keyauthenticator_token.png
  12. Paste your copied key into the TOTP seed field in KeePass then click next until the end. You don’t need to enter a server to sync with on the next screen. keepass_totp_seed.png
  13. Now you need a way to view the MFA generated key. In the KeePass app click the View menu, then click Configure Columns. Scroll to the bottom of the window and check the option for TOTP then click OK.configure_totp.png
  14. Now in your KeePass view you will have a column labeled TOTP that will have an MFA key that changes every 30 seconds.keepass_totpview.png
  15. In order for your keys to become effective with Google, you need to take the current key and enter it into the Google website where you got your TOTP seed from. To do this you can right click on the entry in KeePass and click Copy TOTP.copy_totp.png
  16. Go back to your browser, click next, paste the copied key into the text box then click verify.authenticator_verify.png
  17. You now have your MFA/TOTP/2 Step key available to you within KeyPass, and your smart phone if you setup your Google Authenticator app in step 9.

Conclusion

I have been very happy with KeePass. It’s free, it’s secure, it’s pretty straight forward, and it integrates well into my daily habits. Feel free to leave any suggestions for me below!