Category Archives: Programming

Update to Batch Converting Visio Files

After some trail and error from my previous post we went through batch converting 300+ Visio vsd files over to vsdx. Overall the files size was reduced by 70%, dropping from over 6 Gb to around 2Gb, and allowing the files to open/save to the network a lot quicker. The only caveat I found was Visio 64-bit is the best way to do this and is most stable, especially with files over 25Mb. Above about 33 – 35Mb and the 32-bit version would randomly crash. With that said I added in some basic error detection to skip corrupt Visio files, some user variables to keep personal info or remove along with deleting the original file or not, and the ability to do sub directories. Here is the modified code:

Public FilesAttempted As Integer
Public FilesConverted As Integer
Public FilesDeleted As Integer
Public FilesSkipped As String

Sub ConvertToVsdx()
FilesAttempted = 0
FilesConverted = 0
FilesDeleted = 0
FilesSkipped = ""
Dim FileSystem As Object
Set FileSystem = CreateObject("Scripting.FileSystemObject")
Dim HostFolder As String
Dim DeleteOriginal As Boolean
Dim RemovePersonal As Boolean

''' HostFolder is directory to start it. Change to your base directory.
HostFolder = "T:\"
''' DeleteOriginal will delete the original file as long as the VSDX was created. Either True or False
DeleteOriginal = True
''' RemovePersonal will remove personal information from the file. Reduces the size a little but you might want to keep the original info
RemovePersonal = False

DoFolder FileSystem.GetFolder(HostFolder), DeleteOriginal
MsgBox "Conversion complete! " & vbCrLf & vbCrLf & "Files attempted: " & FilesAttempted & vbCrLf & "Files converted: " & FilesConverted & vbCrLf & "Files deleted: " & _
FilesDeleted & vbCrLf & "Files with issues: " & vbCrLf & FilesSkipped, vbOKOnly + vbInformation, "Conversion Complete"
End Sub

Sub DoFolder(Folder, DeleteOriginal)
On Error GoTo ErrHandler:
Dim SubFolder
For Each SubFolder In Folder.SubFolders
DoFolder SubFolder, DeleteOriginal
Next
Dim File
For Each File In Folder.Files
' For each file name sure its a vsd and not a temp file
If ((Right(File, 3) = "vsd") And (Right(File, 4) <> "~vsd")) Then
FilesAttempted = FilesAttempted + 1
' Open the file
Application.Documents.Open File
' Remote personal info if set
If RemovePersonal = True Then
Application.ActiveDocument.RemovePersonalInformation = True
End If
  ' Loop through each master then check across pages to see if it is used
  Index = Application.ActiveDocument.Masters.Count
  While Index > 0
  bMasterUsed = False
  Set oMaster = Application.ActiveDocument.Masters.Item(Index)
  For Each oPage In Application.ActiveDocument.Pages
  For Each oShape In oPage.Shapes
  If oMaster.Name = oShape.Name Then
  bMasterUsed = True
  End If
  Next
  Next
  ' if Not used delete it from the document stencil
  If bMasterUsed = False Then
  oMaster.Delete
  End If
  Index = Index - 1
  Wend

  ' Save as a vsdx and increase our counter
  Application.ActiveDocument.SaveAs File & "x"
  Application.ActiveDocument.Close
  FilesConverted = FilesConverted + 1

  ' Delete the original if set and the new vsdx exists
  If ((DeleteOriginal = "True") And (FileExists(File & "x"))) Then
  SetAttr File, vbNormal
  Kill File
  FilesDeleted = FilesDeleted + 1
  End If
NextFile:
End If
Next
Done:
Exit Sub

ErrHandler:
Debug.Print "Error encountered. Error number: " & Err.Number & " - Error description: " & Err.Description
If File <> "" Then
FilesSkipped = FilesSkipped & File & vbCrLf
GoTo NextFile:
End If
End Sub

Function FileExists(ByVal FileToTest As String) As Boolean
FileExists = (Dir(FileToTest) <> "")
End Function

If you use this please let me know how it goes or any tweaks that need to be made.

Batch Converting Visio VSD files to VSDX

I use Visio for my network and building layout drawings for simplicity. We have recently upgraded to Visio 2016 from 2007 and along with that there is a new file format, a .vsdx instead of a .vsd. Now I’m not sure of all the behind the scene changes but I do know that when you check a files properties you can reduce the file size by removing unused shapes, remove personal info, and save your old files as the new format. Between these three options the file size will drop 40 – 70% per file. Now when you have hundreds or even thousands of these you can gain a significant amount of file space back along with quicker overall loading and saving times. The problem is this looked to be a manual process.

So I did some searching and of course others have already tried to script this. I found two main articles, one on batch converting the vsd to vsdx and a second on removing the unused objects. I combined the two into a single script and added in the remove personal information option. So now you can do all three with a single script.

This is in VBA so to use this start out by opening a new blank Visio drawing. Hit ALT+F11 to get into Microsoft Visual Basic for Applications. Right click your drawing on the top left (by default Drawing1) and insert a module. Then copy and paste this in:

Sub ConvertToVsdx()
Dim strPath As String
Dim strFile As String
strPath = "C:\Temp2\"
strFile = Dir(strPath & "*.vsd")
Do While strFile <> ""
If Right(strFile, 3) = "vsd" Then
Application.Documents.Open strPath & strFile
Application.ActiveDocument.RemovePersonalInformation = True
' Loop through each master then check across pages to see if it is used
Index = Application.ActiveDocument.Masters.Count
While Index > 0
bMasterUsed = False
Set oMaster = Application.ActiveDocument.Masters.Item(Index)
For Each oPage In Application.ActiveDocument.Pages
For Each oShape In oPage.Shapes
If oMaster.Name = oShape.Name Then
bMasterUsed = True
End If
Next
Next
' if Not used delete it from the document stencil
If bMasterUsed = False Then
oMaster.Delete
End If
Index = Index - 1
Wend
Application.ActiveDocument.SaveAs strPath & strFile & "x"
Application.ActiveDocument.Close
End If
strFile = Dir
Loop
End Sub

Change the strPath to the directory you want to run this on and hit F5 to run. It will open each drawing with a vsd extension in that directory, set the remove personal information tag, remove the unused objects/stencils, then save it as a vsxd.

I haven’t adapted it to do subfolders yet as this is all I really needed but I’m sure there is a way to iterate through directories.

Compiling NodeMCU ESP32 Firmware

Recently I’ve been doing some experimenting with the NodeMCU ESP8266 and ESP32-S chips to use in home automation, sending data back to my Samsung SmartThings system. The ESP32 is considered a development board and as such not everything works as expected. One of the issues I was having was the ADC isn’t very linear resulting in some of my analog inputs being pretty far off (over 4%). With that said the firmware is being updated pretty regularly so I wanted to compile a new version. I attempted to follow the instructions available (https://nodemcu.readthedocs.io/en/dev-esp32/en/build/) but was having trouble. Finally I created a Ubuntu 16.04.3 LTS virtual machine using VirtualBox but then ran into a bunch of other issues since it didn’t have any of the dependencies installed. Well after some experimenting I got it working and here are the steps:

  1. Install Ubuntu 16 (should work with 17 also)
  2. Update it (search for update and launch the software updater)
  3. Drop to a terminal (CTRL+ALT+T), typing each item, and press enter:
    • sudo apt-get update
    • sudo apt-get upgrade
    • sudo apt-get install libncurses5-dev libncursesw5-dev flex bison gperf python-serial
    • sudo apt-get install git
  4. In the same terminal window grab the firmware code: “git clone --branch dev-esp32 --recurse-submodules https://github.com/nodemcu/nodemcu-firmware.git nodemcu-firmware-esp32
  5. Move to the source directory with “cd nodemcu-firmware-esp32”
  6. Start up the menu using “make menuconfig”

You should now be able to select your config options and save a sdkconfig file that you can use to make the firmware. Now I couldn’t save the sdkconfig in the same directory and ended up saving it to my home directory then making a copy of the modemcu-firmware-esp32 directory, pasting the config in there, and running make from there to compile. I’m assuming this is because I used git to pull down the repo and its read only (compiling from the copy with my sdkconfig worked so I didn’t try to figure it out).

If you want to get fancy you can also share out your USB through VirtualBox and flash the chip from the virtual machine using make flash but I didn’t like that idea so I transferred out the NodeMCU.bin from the build directory to my host machine (Windows 10) and used NodeMCU-PyFlasher-2.0 to flash to firmware. Afterward I found that my ADC’s, while not 100% accurate, were a lot closer. Hopefully they keep making progress on this chip as it seems like a very capable replacement to the ESP8266.

VB.Net Drag and Drop from Outlook

One thing I have been working on for years now is a database program that among other things has a part that lets you attach files into it. The drag and drop functionality was easy for regular files on the hard drive but a feature request came in to drag and drop files from a outlook attachment. Luckily someone had already figured this out in C# which I converted into VB here: http://www.codeproject.com/Articles/7140/Drag-and-Drop-Attached-File-From-Outlook-and-ab. This was all fine and dandy until I got a request last week to attach the actual email itself, not a attachment. After more searching I found someone that did this using the Outlook interop libraries here: http://www.emoreau.com/Entries/Articles/2008/05/Dropping-a-Outlook-message-on-your-application.aspx. So after a little more playing I updated my drag and drop code to decipher between a normal file, a Outlook attachment, or a Outlook email and act accordingly.

Couple notes: DisplayMessageBox is a custom message box used by my program (replace with MsgBox if needed) and AddTempFileToArray does exactly that, adds any temp files I create or use to a array which I then delete when my program closes (don’t want a bunch of temp files created and left), and SaveButton is enabled only during a edit operation.

In my control where I will accept the drop:

Private Sub MyControlToAcceptTheDrop_DragEnter(ByVal sender As Object, ByVal e As System.Windows.Forms.DragEventArgs) Handles MyControlToAcceptTheDrop.DragEnter
' Make sure that the format is a file drop.
If (e.Data.GetDataPresent(DataFormats.FileDrop)) And (SaveButton.Visible = True) Then
e.Effect = DragDropEffects.Copy
ElseIf (e.Data.GetDataPresent("FileGroupDescriptor")) And (SaveButton.Visible = True) Then
e.Effect = DragDropEffects.Copy
Else
' Do not allow drop.
e.Effect = DragDropEffects.None
End If
End Sub

And my code to handle the drop:

'''

''' Handle File Drops
'''

''' DragEventArgs ''' Path to the actual file or temp file
''' Returns the full path to the file being dropped or to a temp file that contains the file in memory (for use with Outlook or other program drag drops)
Friend Function HandleFileDrops(ByVal e As System.Windows.Forms.DragEventArgs) As String
Try
If e.Data.GetDataPresent(DataFormats.FileDrop) Then
' We have a file so lets pass it to the calling form
Dim Filename As String() = CType(e.Data.GetData(DataFormats.FileDrop), String())
HandleFileDrops = Filename(0)
ElseIf e.Data.GetDataPresent("FileGroupDescriptor") Then
' We have a embedded file. First lets try to get the file name out of memory
Dim theStream As Stream = CType(e.Data.GetData("FileGroupDescriptor"), Stream)
Dim fileGroupDescriptor(512) As Byte
theStream.Read(fileGroupDescriptor, 0, 512)
Dim fileName As System.Text.StringBuilder = New System.Text.StringBuilder("")
Dim i As Integer = 76
While Not (fileGroupDescriptor(i) = 0)
fileName.Append(Convert.ToChar(fileGroupDescriptor(i)))
System.Math.Min(System.Threading.Interlocked.Increment(i), i - 1)
End While
theStream.Close()
' We should have the file name or if its a email the subject line. Create our temp file based on the temp path and this info
Dim myTempFile As String = Path.GetTempPath & fileName.ToString
' Look to see if this is a email message. If so save that temporarily and get the temp file.
If InStr(myTempFile, ".msg") > 0 Then
Dim objOL As New Microsoft.Office.Interop.Outlook.Application
Dim objMI As Microsoft.Office.Interop.Outlook.MailItem
If objOL.ActiveExplorer.Selection.Count > 1 Then
DisplayMessageBox("You can only drag and drop one item at a time into this screen. The first item you selected will be used.", "One Item At A Time", , FormStartPosition.CenterParent)
End If
For Each objMI In objOL.ActiveExplorer.Selection()
objMI.SaveAs(myTempFile)
Exit For
Next
objOL = Nothing
objMI = Nothing
Else
' If its a attachment we need to pull the file itself out of memory
Dim ms As MemoryStream = CType(e.Data.GetData("FileContents", True), MemoryStream)
Dim FileBytes(CInt(ms.Length)) As Byte
' read the raw data into our variable
ms.Position = 0
ms.Read(FileBytes, 0, CInt(ms.Length))
ms.Close()
' save the raw data into our temp file
Dim fs As FileStream = New FileStream(myTempFile, FileMode.OpenOrCreate, FileAccess.Write)
fs.Write(FileBytes, 0, FileBytes.Length)
fs.Close()
End If
' Make sure we have a actual file and also if we do make sure we erase it when done
If File.Exists(myTempFile) Then
' Assign the file name to the add dialog
HandleFileDrops = myTempFile
Call AddTempFileToArray(myTempFile)
Else
HandleFileDrops = String.Empty
End If
Else
Throw New System.Exception("An exception has occurred.")
End If
Catch ex As Exception
DisplayMessageBox("Could not copy file from memory. Please save the file to your hard drive first and then retry your drag and drop.", "Drag and Drop Failed")
HandleFileDrops = String.Empty
End Try

End Function

As you might be able to guess I only look at the first email message if multiple are selected and let the user know that also. I’m sure there is a way to loop through multiple files, Outlook attachments, or Outlook emails but I don’t need that functionality so I didn’t code it (but I did warn in case it happens).