Category Archives: Programming

Setting up a Raspberry Pi Zero W for Pi-hole

Programming, , , ,

I’ll start by saying there are lots of guides to do this already but none of them worked for me from beginning to end and I pulled parts from different ones to get things working. So after some trial and error I wrote my own guide. Some other sites I used as reference: A good quick start guide from Reddit, Info on changing the locale and keyboard layout for US, and Setting up a headless Pi Zero. I’m also assuming if you are reading this article that you already know what Pi-hole is and want to implement it on your network.

First things first. You will need a Pi Zero W (if you have a local MicroCenter they have them in store for $5 pretty often), a 8Gb micro SD card, a micro usb cable to power the zero, and a way to write to the SD card, either a SD card adapter or your computer if it has a slot for it. You may also want a case to put yours in if you plan on mounting it and don’t want to short anything out (my personal favorite that runs about $6). If you want to actually see whats going on you will also need a mini HDMI to HDMI cable although hopefully you don’t need to do that. And if you want to get really fancy you can get a Pi Zero W with header pins (or solder some on yourself) and a display such as the Adafruit one to show the IP address and some status info.

Lets grab the software you’ll need:

Then put it together:

  • Format the SD card using the SD Card Formatter utility.
  • Using balenaEtcher burn the Raspbian image to the SD card. By default it will “eject” the SD card once its done. You also might get Windows warnings about a problem with the disk, ignore these and close/cancel the warnings.
  • Pop out the SD card and put it back in. You might get the same warnings, again ignore them.
  • Using Notepad (or Notepad++) create a file named “ssh” with no extension and a single space in it. Save that to the root of the SD card along with the other files.
  • Again using a notepad program create a file called “wpa_supplicant.conf” with the following text (using your country code, SSID, and password) and save that to the root of the SD card. If your SSID is being broadcast you do not need the scan_ssid line as it will slow down the connecting slightly although it doesn’t hurt to have it in there (see here for more info on this file):
country=US
ctrl_interface=DIR=/var/run/wpa_supplicant GROUP=netdev
update_config=1

network={
  ssid="YourSSID"
  scan_ssid=1
  psk="YourPassword"
}
  • You should have everything on the SD card needed to boot up and connect to your WiFi. Properly eject the SD card from your computer, pop it into the Pi, and boot it up.
  • Give it 90 – 120 seconds to boot. Since this guide is for getting Pi-Hole setup you will want to have a static IP address for your Pi. Go into your router and find out what the current IP address is and also set a reservation or lock the current IP address to your device.
  • Until it reboots it will be at whatever IP it was assigned. Start up PuTTY and put in the current IP address and connect. You will get a security warning, click Yes to accept the keys.

Your Pi is now on your network so the hard part is done. Next is to configure some basic settings and install Pi-hole

  • Login using the default user “pi” and password “raspberry”
  • Now by default the Pi is setup for the EN_GB locale which means your @ symbol and some others aren’t where they normally are which caused me some issues. So lets tell the Pi to add the EN_US locale and change our keyboard layout.
  • Type in sudo raspi-config and enter
    • Select option 4 for Localization then the first option for Locale
    • In the list scroll down and select en_US.UTF-8 UTF-8 by hitting the space bar to select it then enter
    • It will ask you to select a default language for the system. Select en_US.UTF-8 and enter. Give it a minute while it makes the change.
    • At this point for me I tried to change the keyboard layout but it didn’t work. It seems the Pi needs to be rebooted for these changes to take effect so first get out of the config menu by hitting the right arrow twice and selecting Finish
  • Back at the prompt type in sudo reboot. You will be disconnected, give the Pi a minute or so to reboot. Remember if you set a DHCP reservation the IP address may change when it comes back up.
  • Connect with PuTTY and login again. Easiest way is to click the PuTTY icon in the top left corner of the program and select “Restart Session” then log back in or just close and reopen the software.
  • Go back into the config with sudo raspi-config and enter
    • Select option 4 then 2 for change timezone. Select your country and city for your timezone (US -> Eastern for example).
    • Select option 4 again then 3 for the keyboard. In the list you should have a Generic 104-key PC option. Select that then English (US) or whatever keyboard layout you’d prefer. If you don’t see English (US) click Other first then it should be in the list. Hit enter to select then select the actual keyboard layout. You will be asked a couple more questions, select the appropriate answer (usually the defaults).
    • You can also go into the same menu to select US as your wireless country.
  • Now would also be a good time to change your password to something other then the default raspberry. Select the first option and enter a new password.
  • Select Finish to exit the config menu.

So the next recommended thing to do is update the Pi to the latest software:

  • At the terminal prompt type sudo apt-get update -y and enter. It might take a minute or two.
  • Next type sudo apt-get upgrade -y and enter. Chances are this will take 10 – 20 minutes as updates are downloaded and installed.

Once the updates are done reboot again with sudo reboot, log back in using PuTTY, and install Pi-hole:

  • At the terminal prompt type in curl -sSL https://install.pi-hole.net | bash and enter

Go through the prompts and answer as you want it setup. Note the admin webpage password at the end as you will use that to login at http://YourPiAddress/admin. Lastly set your router DHCP to hand out the Pi-hole IP address for DNS instead of itself and you should be all set.

Update to Batch Converting Visio Files

Programming, , , ,

After some trail and error from my previous post we went through batch converting 300+ Visio vsd files over to vsdx. Overall the files size was reduced by 70%, dropping from over 6 Gb to around 2Gb, and allowing the files to open/save to the network a lot quicker. The only caveat I found was Visio 64-bit is the best way to do this and is most stable, especially with files over 25Mb. Above about 33 – 35Mb and the 32-bit version would randomly crash. With that said I added in some basic error detection to skip corrupt Visio files, some user variables to keep personal info or remove along with deleting the original file or not, and the ability to do sub directories. Here is the modified code:

Public FilesAttempted As Integer
 Public FilesConverted As Integer
 Public FilesDeleted As Integer
 Public FilesSkipped As String

 Sub ConvertToVsdx()
   FilesAttempted = 0
   FilesConverted = 0
   FilesDeleted = 0
   FilesSkipped = ""
   Dim FileSystem As Object
   Set FileSystem = CreateObject("Scripting.FileSystemObject")
   Dim HostFolder As String
   Dim DeleteOriginal As Boolean
   Dim RemovePersonal As Boolean

   ''' HostFolder is directory to start it.  Change to your base directory.
   HostFolder = "T:\"
   ''' DeleteOriginal will delete the original file as long as the VSDX was created.  Either True or False
   DeleteOriginal = True
   ''' RemovePersonal will remove personal information from the file.  Reduces the size a little but you might want to keep the original info
   RemovePersonal = False

   DoFolder FileSystem.GetFolder(HostFolder), DeleteOriginal
   MsgBox "Conversion complete! " & vbCrLf & vbCrLf & "Files attempted: " & FilesAttempted & vbCrLf & "Files converted: " & FilesConverted & vbCrLf & "Files deleted: " & _
         FilesDeleted & vbCrLf & "Files with issues: " & vbCrLf & FilesSkipped, vbOKOnly + vbInformation, "Conversion Complete"
 End Sub
 
Sub DoFolder(Folder, DeleteOriginal)
     On Error GoTo ErrHandler:
     Dim SubFolder
     For Each SubFolder In Folder.SubFolders
         DoFolder SubFolder, DeleteOriginal
     Next
     Dim File
     For Each File In Folder.Files
         ' For each file name sure its a vsd and not a temp file
         If ((Right(File, 3) = "vsd") And (Right(File, 4) <> "~vsd")) Then
             FilesAttempted = FilesAttempted + 1
             ' Open the file
             Application.Documents.Open File
             ' Remote personal info if set
             If RemovePersonal = True Then
                 Application.ActiveDocument.RemovePersonalInformation = True
             End If
            ' Loop through each master then check across pages to see if it is used         
            Index = Application.ActiveDocument.Masters.Count
            While Index > 0
              bMasterUsed = False
              Set oMaster = Application.ActiveDocument.Masters.Item(Index)
                For Each oPage In Application.ActiveDocument.Pages
                  For Each oShape In oPage.Shapes
                    If oMaster.Name = oShape.Name Then
                      bMasterUsed = True
                    End If
                  Next
                Next
                ' if Not used delete it from the document stencil
                If bMasterUsed = False Then
                  oMaster.Delete
                End If
                Index = Index - 1
              Wend

              ' Save as a vsdx and increase our counter
              Application.ActiveDocument.SaveAs File & "x"
              Application.ActiveDocument.Close
              FilesConverted = FilesConverted + 1

              ' Delete the original if set and the new vsdx exists
              If ((DeleteOriginal = "True") And (FileExists(File & "x"))) Then
                SetAttr File, vbNormal
                Kill File
                FilesDeleted = FilesDeleted + 1
              End If
NextFile:
        End If
    Next
Done:
    Exit Sub

ErrHandler:
Debug.Print "Error encountered. Error number: " & Err.Number & " - Error description: " & Err.Description
  If File <> "" Then
    FilesSkipped = FilesSkipped & File & vbCrLf
    GoTo NextFile:
  End If
End Sub

Function FileExists(ByVal FileToTest As String) As Boolean
    FileExists = (Dir(FileToTest) <> "")
End Function

If you use this please let me know how it goes or any tweaks that need to be made.

Batch Converting Visio VSD files to VSDX

Programming, , , ,

I use Visio for my network and building layout drawings for simplicity. We have recently upgraded to Visio 2016 from 2007 and along with that there is a new file format, a .vsdx instead of a .vsd. Now I’m not sure of all the behind the scene changes but I do know that when you check a files properties you can reduce the file size by removing unused shapes, remove personal info, and save your old files as the new format. Between these three options the file size will drop 40 – 70% per file. Now when you have hundreds or even thousands of these you can gain a significant amount of file space back along with quicker overall loading and saving times. The problem is this looked to be a manual process.

So I did some searching and of course others have already tried to script this. I found two main articles, one on batch converting the vsd to vsdx and a second on removing the unused objects. I combined the two into a single script and added in the remove personal information option. So now you can do all three with a single script.

This is in VBA so to use this start out by opening a new blank Visio drawing. Hit ALT+F11 to get into Microsoft Visual Basic for Applications. Right click your drawing on the top left (by default Drawing1) and insert a module. Then copy and paste this in:

Sub ConvertToVsdx()
 Dim strPath As String
 Dim strFile As String
 strPath = "C:\Temp2\"
 strFile = Dir(strPath & "*.vsd")
 Do While strFile <> ""
   If Right(strFile, 3) = "vsd" Then
     Application.Documents.Open strPath & strFile
     Application.ActiveDocument.RemovePersonalInformation = True
     ' Loop through each master then check across pages to see if it is used
     Index = Application.ActiveDocument.Masters.Count
     While Index > 0
       bMasterUsed = False
       Set oMaster = Application.ActiveDocument.Masters.Item(Index)
       For Each oPage In Application.ActiveDocument.Pages
         For Each oShape In oPage.Shapes
           If oMaster.Name = oShape.Name Then
             bMasterUsed = True
           End If
         Next
       Next
       ' if Not used delete it from the document stencil
       If bMasterUsed = False Then
         oMaster.Delete
       End If
       Index = Index - 1
     Wend
   Application.ActiveDocument.SaveAs strPath & strFile & "x"
   Application.ActiveDocument.Close
 End If
 strFile = Dir
 Loop
End Sub

Change the strPath to the directory you want to run this on and hit F5 to run. It will open each drawing with a vsd extension in that directory, set the remove personal information tag, remove the unused objects/stencils, then save it as a vsxd.

I haven’t adapted it to do subfolders yet as this is all I really needed but I’m sure there is a way to iterate through directories.

Compiling NodeMCU ESP32 Firmware

Programming, , ,

Recently I’ve been doing some experimenting with the NodeMCU ESP8266 and ESP32-S chips to use in home automation, sending data back to my Samsung SmartThings system. The ESP32 is considered a development board and as such not everything works as expected. One of the issues I was having was the ADC isn’t very linear resulting in some of my analog inputs being pretty far off (over 4%). With that said the firmware is being updated pretty regularly so I wanted to compile a new version. I attempted to follow the instructions available (https://nodemcu.readthedocs.io/en/dev-esp32/en/build/) but was having trouble. Finally I created a Ubuntu 16.04.3 LTS virtual machine using VirtualBox but then ran into a bunch of other issues since it didn’t have any of the dependencies installed. Well after some experimenting I got it working and here are the steps:

  1. Install Ubuntu 16 (should work with 17 also)
  2. Update it (search for update and launch the software updater)
  3. Drop to a terminal (CTRL+ALT+T), typing each item, and press enter:
    • sudo apt-get update
    • sudo apt-get upgrade
    • sudo apt-get install libncurses5-dev libncursesw5-dev flex bison gperf python-serial
    • sudo apt-get install git
  4. In the same terminal window grab the firmware code: “git clone --branch dev-esp32 --recurse-submodules https://github.com/nodemcu/nodemcu-firmware.git nodemcu-firmware-esp32“
  5. Move to the source directory with “cd nodemcu-firmware-esp32”
  6. Start up the menu using “make menuconfig”

You should now be able to select your config options and save a sdkconfig file that you can use to make the firmware. Now I couldn’t save the sdkconfig in the same directory and ended up saving it to my home directory then making a copy of the modemcu-firmware-esp32 directory, pasting the config in there, and running make from there to compile. I’m assuming this is because I used git to pull down the repo and its read only (compiling from the copy with my sdkconfig worked so I didn’t try to figure it out).

If you want to get fancy you can also share out your USB through VirtualBox and flash the chip from the virtual machine using make flash but I didn’t like that idea so I transferred out the NodeMCU.bin from the build directory to my host machine (Windows 10) and used NodeMCU-PyFlasher-2.0 to flash to firmware. Afterward I found that my ADC’s, while not 100% accurate, were a lot closer. Hopefully they keep making progress on this chip as it seems like a very capable replacement to the ESP8266.

VB.Net Drag and Drop from Outlook

Programming, , , , , ,

One thing I have been working on for years now is a database program that among other things has a part that lets you attach files into it. The drag and drop functionality was easy for regular files on the hard drive but a feature request came in to drag and drop files from a outlook attachment. Luckily someone had already figured this out in C# which I converted into VB here: http://www.codeproject.com/Articles/7140/Drag-and-Drop-Attached-File-From-Outlook-and-ab. This was all fine and dandy until I got a request last week to attach the actual email itself, not a attachment. After more searching I found someone that did this using the Outlook interop libraries here: http://www.emoreau.com/Entries/Articles/2008/05/Dropping-a-Outlook-message-on-your-application.aspx. So after a little more playing I updated my drag and drop code to decipher between a normal file, a Outlook attachment, or a Outlook email and act accordingly.

Couple notes: DisplayMessageBox is a custom message box used by my program (replace with MsgBox if needed) and AddTempFileToArray does exactly that, adds any temp files I create or use to a array which I then delete when my program closes (don’t want a bunch of temp files created and left), and SaveButton is enabled only during a edit operation.

In my control where I will accept the drop:

Private Sub MyControlToAcceptTheDrop_DragEnter(ByVal sender As Object, ByVal e As System.Windows.Forms.DragEventArgs) Handles MyControlToAcceptTheDrop.DragEnter
' Make sure that the format is a file drop.
If (e.Data.GetDataPresent(DataFormats.FileDrop)) And (SaveButton.Visible = True) Then
e.Effect = DragDropEffects.Copy
ElseIf (e.Data.GetDataPresent("FileGroupDescriptor")) And (SaveButton.Visible = True) Then
e.Effect = DragDropEffects.Copy
Else
' Do not allow drop.
e.Effect = DragDropEffects.None
End If
End Sub

And my code to handle the drop:

'''

''' Handle File Drops
'''

''' DragEventArgs ''' Path to the actual file or temp file
''' Returns the full path to the file being dropped or to a temp file that contains the file in memory (for use with Outlook or other program drag drops)
Friend Function HandleFileDrops(ByVal e As System.Windows.Forms.DragEventArgs) As String
Try
If e.Data.GetDataPresent(DataFormats.FileDrop) Then
' We have a file so lets pass it to the calling form
Dim Filename As String() = CType(e.Data.GetData(DataFormats.FileDrop), String())
HandleFileDrops = Filename(0)
ElseIf e.Data.GetDataPresent("FileGroupDescriptor") Then
' We have a embedded file. First lets try to get the file name out of memory
Dim theStream As Stream = CType(e.Data.GetData("FileGroupDescriptor"), Stream)
Dim fileGroupDescriptor(512) As Byte
theStream.Read(fileGroupDescriptor, 0, 512)
Dim fileName As System.Text.StringBuilder = New System.Text.StringBuilder("")
Dim i As Integer = 76
While Not (fileGroupDescriptor(i) = 0)
fileName.Append(Convert.ToChar(fileGroupDescriptor(i)))
System.Math.Min(System.Threading.Interlocked.Increment(i), i - 1)
End While
theStream.Close()
' We should have the file name or if its a email the subject line. Create our temp file based on the temp path and this info
Dim myTempFile As String = Path.GetTempPath & fileName.ToString
' Look to see if this is a email message. If so save that temporarily and get the temp file.
If InStr(myTempFile, ".msg") > 0 Then
Dim objOL As New Microsoft.Office.Interop.Outlook.Application
Dim objMI As Microsoft.Office.Interop.Outlook.MailItem
If objOL.ActiveExplorer.Selection.Count > 1 Then
DisplayMessageBox("You can only drag and drop one item at a time into this screen. The first item you selected will be used.", "One Item At A Time", , FormStartPosition.CenterParent)
End If
For Each objMI In objOL.ActiveExplorer.Selection()
objMI.SaveAs(myTempFile)
Exit For
Next
objOL = Nothing
objMI = Nothing
Else
' If its a attachment we need to pull the file itself out of memory
Dim ms As MemoryStream = CType(e.Data.GetData("FileContents", True), MemoryStream)
Dim FileBytes(CInt(ms.Length)) As Byte
' read the raw data into our variable
ms.Position = 0
ms.Read(FileBytes, 0, CInt(ms.Length))
ms.Close()
' save the raw data into our temp file
Dim fs As FileStream = New FileStream(myTempFile, FileMode.OpenOrCreate, FileAccess.Write)
fs.Write(FileBytes, 0, FileBytes.Length)
fs.Close()
End If
' Make sure we have a actual file and also if we do make sure we erase it when done
If File.Exists(myTempFile) Then
' Assign the file name to the add dialog
HandleFileDrops = myTempFile
Call AddTempFileToArray(myTempFile)
Else
HandleFileDrops = String.Empty
End If
Else
Throw New System.Exception("An exception has occurred.")
End If
Catch ex As Exception
DisplayMessageBox("Could not copy file from memory. Please save the file to your hard drive first and then retry your drag and drop.", "Drag and Drop Failed")
HandleFileDrops = String.Empty
End Try

End Function

As you might be able to guess I only look at the first email message if multiple are selected and let the user know that also. I’m sure there is a way to loop through multiple files, Outlook attachments, or Outlook emails but I don’t need that functionality so I didn’t code it (but I did warn in case it happens).