Google Protocol Buffers Python Implementation

I vaguely recall hearing about Google’s Protocol Buffers a while ago, but I forgot about them until I recently encountered a project using them. Since I wanted to use the data from the project, it seemed that I would have to learn how to use Protocol Buffers.

After looking at Google developer docs for a while, I settled on the Python tutorial at https://developers.google.com/protocol-buffers/docs/pythontutorial.

I’m working on a Windows 10 computer, but using Ubuntu on Windows so I already had an environment with Python 2.7.x installed.

The tutorial is pretty good, but it makes certain assumptions. I didn’t build the compiler from source. I installed a pre-compiled binary from https://github.com/google/protobuf/releases. This was the file: https://github.com/google/protobuf/releases/download/v3.4.0/protoc-3.4.0-linux-x86_64.zip.

I downloaded the .proto file from the project on which I’m working, and ran ./protoc –python_out OUT_DIR project.proto. That generated the Python code for interacting with that particular data structure.

I tried importing that into my project.py file, but it complained: “ImportError: No module named google.protobuf”. If I had followed the tutorial more closely, I think the assumption is that you’ll build the compiler from source and that you’ll also install your own protobuf runtime libraries. I could’ve done that, but I was lazy so I just installed pip (apt-get install python-pip) and then ran “pip install protobuf”.

According to some Google docs, the runtime library version should be the same as the compiler version, which makes sense. pip had version 3.4.0 which was the same version as the pre-compiled compiler binary I downloaded so that was handy.

Now my Google Protocol Buffer generated Python module is loading, so I’m off to try it out. I think the hardest bit is behind me now.

I’m actually really excited, because the project using the Protocol Buffers is a Java project, but I want a Python tool for interacting with the data from that Java project, and this should work pretty seamlessly. It seems like there are actually a lot of runtime implementations available for Protocol Buffers, so this would be a nice way of sharing data among a number of projects.

I wonder what sort of uptake Protocol Buffers see outside of Google. It seems that Google uses them a lot, but this project is the first time I’ve encountered them in the wild, and I think Protocol Buffers have been in the public domain for about a decade now. In some ways, they’re not as convenient as serializing as JSON or XML, but in other ways they seem a million times better.

I suppose I’ll form my ultimate opinions after I have some experience working with them. I’m intrigued so far though!

 

Advertisements

Linux Subsystem on Windows 10: Python and Ansible

Last week, using a couple of sets of instructions, I installed the Linux Subsystem on Windows 10. You can find the instructions at https://www.jeffgeerling.com/blog/2017/using-ansible-through-windows-10s-subsystem-linux#comment-6341 and https://msdn.microsoft.com/en-au/commandline/wsl/install_guide.

Since then, I haven’t had much of a chance to play with it. However, I’ve been meaning to use Python more often in my daily work, so I thought it would be fun to write a Python script and run it in “Bash on Ubuntu on Windows”.

But first… I wanted to set up my development environment in a way that would be easy to keep track of. I think it’s really easy to get ahead of ourselves sometimes and just start downloading a million packages using a few different package managers and then we wind up with a system full of stuff that we don’t actually use. Of course, virtual environments and the like can get you around that a bit at a language level, but I like documentation… and I really like Ansible.

Unlike Jeff Geerling in the above instructions, I just ran “apt-get install ansible” to install Ansible via “Bash on Ubuntu on Windows”. I’m not too fussed about getting the most recent version, so the version in Ubuntu’s Xenial repositories from January 2016 (ie 2.0.0.2) suits me fine for experimenting at this point.

At this point, I’ve been using Ansible for years, so I put together a quick little playbook to install the package “python3-mysqldb” and that ran successfully. While Ansible itself uses python2, “Bash on Ubuntu on Windows” comes with python3 out of the box. I’ve used python2 a bit over the years, both for developing Ansible modules and for other Python scripts, but I thought I’d try something new. Plus, if I’m going to be using Python more and more, I may as well use the current version rather than the legacy version of Python.

I don’t have much else to say at this point. If I keep experimenting with “Bash on Ubuntu on Windows”, I’ll make more follow-up posts. Honestly, I should be doing more work directly on Windows, as I want to learn more about that, since I’m already familiar with Linux servers and Linux desktops/laptops from my personal and professional life. But… being able to use Linux from within Windows seems so convenient! Of course, I could be running Python on Windows directly, but Windows often seems like an unpleasant place to work. It feels like it’s more difficult to keep track of things there. But maybe that’s just due to my unfamiliarity with working on Windows as a developer. Maybe I’ll write some Python tools and run them from Windows and from Ubuntu on Windows and see if I can get used to using Windows as a developer…

Bitten by Microsoft Windows 10 Updates

Earlier in the week, when I was turning off my work computer, I selected the “Update and restart” option.

This morning I turned on the computer before commencing my morning routine, so that it would be booted up and ready to go for 9am.

At 9am, I found it was about 12% through the updates, and the screen advised me that it would “take a while”.

Sure enough, in the end, it took about 1.5 hours to install all the updates for Windows 10.

I was overjoyed to finally have access to my desktop…until I noticed the resolution was completely wrong and my dual monitors which are usually extended were actually mirrored.

Readers of this blog may remember that I had to play around with the drivers for the ATI Radeon HD 4250 in this computer: https://tech4lib.wordpress.com/2016/02/03/windows-10-pro-and-legacy-amd-drivers/

So I hit “Win + R”, noticed “devmgmt.msc” was still there from the last time I went through this process, and hit Enter.

As per my blog post from February, I expanded the “Display adapters” section, right-clicked on “ATI Radeon HD 4250”, and noted that “Device status” read that there were no drivers installed. Fun…

Skimming through my old blog post, I switched to the “Microsoft Basic Adapter”, which in hindsight I probably didn’t need to do but I’m documenting here anyway since I did it. In Device Manager, I noticed my “ATI Radeon HD 4250” adapter had been replaced with “Microsoft Basic Adapter”.

Then I re-ran the executable “13-1-legacy_vista_win7_win8_64_dd_ccc.exe” which was still in my Download directory from February. It said that the ADMD Catalyst Control Centre was already installed, but that it would happily install it again if I clicked the button.

Moments pass, my monitors flicker, and voila! My extended display is back! I went back into Device Manager, noticed that “ATI Radeon HD 4250” was back as the Display adapter, and reviewed the Events tab.

  • 9:21: Device install requested
  • 9:22: Device install requested
  • 9:24: Device migrated
  • 9:24: Device installed (c0296217.inf)
  • 10:14: Driver service added (BasicDisplay)
  • 10:14: Device installed (display.inf)
  • 10:16: Driver service added (amdkmdap)
  • 10:16: Driver service added (amdkmdap)
  • 10:16: Driver service added (AMD External Ev…)
  • 10:16: Device installed (c8160540.inf)

Looking back at the blog post from February, that’s pretty much exactly what happened back then too.

I couldn’t find c0296217.inf on my system, so no wonder Device Manager said there was no driver installed. And c8160540.inf can be found at C:\AMD\Support\13-1-legacy_vista_win7_win8_64_dd_ccc\Packages\Drivers\Display\W86A_INF\C8160540.inf

I’m certainly no pro when it comes to drivers and hardware, but I’m glad that I was able to make it work again!

I have another work system which has some third-party driver adapter drivers, and Windows Updates almost always cause them to be uninstalled, and then I have to re-install them to re-gain the desired dual-monitor functionality.

In any case, I can finally start work this morning…

Troubleshooting network adapters on Windows

We all love the Internet and connectivity, but how often do we think about “how” we connect?

At work, I use a Windows PC, which has a physical Ethernet network adapter and a physical WiFi network adapter. Typically, I just use the Ethernet adapter, as it’s faster. I don’t need my desktop to use WiFi, unless I’m having to manage something on our wireless network. I recently noticed the desktop also has a Bluetooth adapter, but I haven’t used that yet, although an increasing interest in the Internet of Things means that I might start to use it soon!

In any case, that’s 3 network adapters. Already it’s starting to sound like a lot. But I also have a host-only virtual adapter for Virtualbox, so that I can have a private connection with my Virtualbox virtual machines. That’s 4 adapters now.

I also use a USB Maxon Modmax modem with a SIM card for connecting to a private network. That’s 5 adapters.

Recently, I’ve also started to use VMware on the same PC. VMware installs another 2 virtual adapters (host-only and NAT). That’s 7 adapters.

After installing VMware, I started to have issues with the USB modem, so I disabled those adapters, and everything was fine…

…until I wanted to use the USB modem in the VMware virtual machines, and to do that I needed the VMware virtual NAT adapter. So I tried that again and had the following result: “C-motech UI Main has stopped working.”

c-motech-crash

The Modmax connection would stay active until I dealt with that C-motech UI Main window. As soon as I clicked “Debug”, “Close program” or “X”, it was all over.

So I could have my VMware adapter and Modmax adapter enabled and achieve the network functionality I required… but I was blighted by this pop-up window that I had to try and hide on my screen. This wasn’t going to be a long-term thing I wanted to live with.

 

I called Maxon technical support to see if they had any insight, but they didn’t. The Modmax was also end-of-life years ago, so they weren’t going to work on troubleshooting it too thoroughly, which was fair enough.

So I decided to experiment a bit by disabling some other adapters and trying again… and voila it worked! I realized that the Modmax modem would work, so long as there was a maximum of 5 enabled network adapters (including its own adapter).

I can’t explain why this is the case. I’m guessing it’s a bug in the Modmax Connection Manager, but maybe Windows has some secret limitation… who knows? At the end, it matters that it works.

I rang Maxon back up to share my discovery, and that’s that. I’m writing this post in the event that someone else is Googling “C-motech UI Main has stopped working”. Hopefully my efforts will help them avoid this crash!

Getting started with Arduino

I think I first heard about Arduino in 2012. I was studying for my masters in library and information studies, and I was working as a systems librarian. While I found the idea of working with hardware interesting, my main focus was software and servers.

But in 2015 an engineer I know gave me an Arduino Uno. Not knowing what to do with it, I put it in a drawer until 2016 when I taught myself C, read an official tutorial online, and discovered my printer’s USB-B to USB-A cable could be used to provide power and code for the board.

I stated with the blink tutorial then made it send a SOS signal with the on board LED. Then another engineer challenged me to dim a LED he gave me, so I plugged it into the board (without a resistor which is naughty but worked fine), and started playing with PWM (pulse width modulation).

I was running out of things I could do, so I started reading as much as I could. I thought about sourcing a breadboard, LEDs, LCDs, piezos, pots, push buttons, motors, etc. I realised though that it would be faster, easier, and probably cheaper just to buy an Arduino Starter Kit.

A few days later, it arrived and I examined all my goodies. To date, I’ve only had time for the first two projects, which are just about using switches to light up LEDs. But I’m looking forward to playing with the motors, LCD, piezo, temperature sensor, pots, and photosensitive resisters with the other projects.

I like the idea of home automation. A robot greeter using PIR (passive infrared) motion detectors; using Bluetooth to signal my arrival home and kick off some automated processes like phone backups; using IR to signal household devices instead of using lots of different remotes. If the temperature drops, turn on a heater; hacking a doorbell to alert a device which sends me an email or turns on a door cam to see who is there.

I’d like to fit a 3G module to a board so that I could communicate with it remotely, but I don’t know what my device would need to do remotely. Such devices are useful to government and corporations, but what use are they to an individual? I suppose if I was a farmer, it could be used for measuring rainfall or temperature. I suppose you could make a long-range robot. A 3G module enables so much more physical freedom than a device controlled by Bluetooth or WiFi.

But what does all this have to do with libraries? You could use an Arduino or Raspberry Pi to make your own self-checkout or security gate, but those products already exist.

I figure my newfound knowledge of electronics would be of more use in a library makerspace.

I sometimes wonder why we have makerspaces in libraries, but it seems a natural evolution. Libraries have traditionally been about the mental world, but now they’re branching out into the physical world too. Marrying the two together with information resources and physical tools.

Most of my project ideas are for home, but I’m sure people have all sorts of ideas for their art, their businesses, their inventions, their own diverse lives.

I think libraries provide a collection of specialized resources we couldn’t afford ourselves, and they also serve as a place for expanding our collective knowledge and skills. As we become digital citizens, we should become closer to the art and craft that comprises our world.

Politicians talk about innovation… and I reckon that’s how you do it. You make spaces for information exchange and promote creativity.

Anyway, I’m just getting started with Arduino, but I’m excited by where I’m going.

Lessons in C: teaching myself pointers and pointer arithmetic

First of all, I love the openness of open source. When you’re a curious person, you’re able to actually find the reasons why the tools you’re using behave in the way that they do. 
Second, I love that I can decide to teach myself how to program in C, and then just go off and do it. The Internet is a wonderful resource, basic hardware is available cheaply, and free software like the free GNU Compiler Collection (GCC) means that you can code without opening your wallet.
 
Long story short, I was writing a C function to read arbitrarily long lines from a file (ie a function to automatically reallocate memory to the buffer as needed whilst reading from the file), and the snippets I was looking at online showed that people were doing things like “fgets(buffer + last, length, file)”. I found “buffer + last” to be utterly confusing.
 
“last” is an integer while “buffer” is a char pointer, which starts to make sense when you look into pointer arithmetic and the fgets source code: http://mirror.fsf.org/pmon2000/3.x/src/lib/libc/fgets.c
 
It all seemed straight forward enough to me except for two lines:
 
*p++ = c;
*p = 0;
 
“c”is a character obtained from a file via fgetc, and “p” is a copy of the buffer pointer.
 
It was clear that “*p++ = c;” was adding the “c” character onto the character array, but how? Well, according to the order of operations, it was really doing “*p = c; p++”, which meant it was dereferencing the “p” pointer and assigning the c variable value to that memory location. Then, it was changing the pointer to point to the next memory address.
 
That’s where *p = 0 comes in. We’ve moved on to the next address via that last “p++” so now we’re setting the next bit of memory to 0, which – if we look at our ascii table at http://www.asciitable.com/ – is decimal for NULL. Since C strings are null terminated, we’re indicating that’s the end of our string.
 
I found it confusing that when I printed “buffer” after the fgets() that it was showing the full string since I thought that I had moved the pointer to a different memory address, but then I realized it was actually the “p” pointer which had been subject to the pointer arithmetic and not the original “buffer” pointer. Of course “buffer” was still pointing to the same place, which allowed me to read the whole string out of the buffer, rather than just the last character assigned.
Now I understand how my C program is reading lines of arbitrary length from a file and printing them out in the terminal window.
With the help of “Valgrind”, I can also see that I’m not leaking memory as well, as I’m making sure to free() my pointers where needed. Really like Valgrind actually, as it also points out other mistakes which don’t necessarily lead to segfaults and such.
Now if I were really keen, I could probably optimise how much memory is reallocated when reading really long lines (in better ways than how others describe online), but there’s probably better ways to spend one’s time.
Now that I better understand pointers, pointer arithmetic, strings, and reading from files, I could in theory create a library for handling MARC records in C, or I could make a really basic encryption program, or both! At the very least, I’ll have a better understanding of how to read the source code of the Zebra indexing engine used by the Koha LMS. In fact, at this point, I might be able to start contributing patches to Zebra!
When I started working on Koha back in 2012, I had never coded in Perl. I had never used Template Toolkit. I had never used Git. I thought Zebra was just an animal, and that Apache referred to helicopters and some people indigenous to America. Now I can use these tools in my sleep. Considering that Perl, Git, and Zebra are all written in C, perhaps this is the next step in understanding those tools, how best to use them, how to fix them, and how to improve them.

Windows 10 Pro and Legacy AMD Drivers

[Preface: I got Windows 10 Pro to detect dual monitors with a AMD ATI Radeon HD 4250 graphic card, which AMD and most other people will say you can’t do.]

Apologies to librarians reading the blog, as I’ve been in a much more technical place lately, but maybe you’ll find the following useful too.

Recently, I’ve been putting together a computer (with a AMD Phenom(tm) II X6 1055T processor) at work, which my boss is going to send home with me, so that I can work remotely from home.

It already had Windows 8.1 running fine on it, but we decided to upgrade to Windows 10 just to be as up-to-date as possible. We packed up the drive with Windows 8.1, cloned Windows 10 onto another drive, plugged it into the machine, turned it on, and wished for the best.

Everything seemed to come up fine, but the display was being duplicated rather than extended across 2 monitors. After booting into Windows 10, the computer also said that it was missing AMD graphical drivers. I’d encountered a similar problem in the past, so I hopped onto AMD’s website (http://support.amd.com/en-us/download), and noticed that they didn’t have any up-to-date drivers for Windows 10 for their ATI Radeon HD 4xxx series. The last Windows operating system I found supported was Windows 8. See the below image for what criteria I used on the AMD website.

amd_download

The more I read online, the more dire it seemed to be. Most people said that you were up the creek without a paddle… some people gave suggestions which didn’t work… and at the end of it all… I figured I’d have a crack at it myself.

I downloaded the AMD driver auto-detect tool, and it said it couldn’t find any drivers for my system. I’d seen such things before and figured I’d discount this as the auto-detect isn’t great.

I downloaded the Radeon HD 4xxx Series PCIe drivers I could find (as you can see in the image above), which claimed to work with Windows Vista, Windows 7, and Windows 8. They came bundled up in an installer (13-1-legacy_vista_win7_win8_64_dd_ccc.exe), so I downloaded the whole thing, and tried installing that and restarting

No luck.

I tried uninstalling the AMD Catalyst Control Center (ie the “ccc” from the .exe installer), restarting, re-installing, restarting… no luck.

I pressed “Win + R” to get a Run prompt, typed in “devmgmt.msc” to get the device manager up. I expanded the “Display adapters” which listed the AMD ATI Radeon HD 4250 device, right-clicked and clicked on Properties, and went to the “Driver” tab.

There I clicked “Update Driver…”, “Browse my computer for driver software”, “Let me pick from a list of device drivers on my computer”, chose “Microsoft Basic Display Adapter”, and clicked “Next”. I restarted the computer, and… it still wasn’t detecting my second monitor.

So I tried uninstalling the AMD Catalyst Control Center again, restarting, reinstalling… and this time I got an extra pop-up during the process. It claimed to be from”Advanced Micro Devices” (ie AMD) and related to the display, so I clicked install, and went back to my existing workstation… when out of the corner of my eye I saw the two monitors flicker. I focused on them and noticed that they were now extended and not duplicated!

I rushed over to the computer and found that AMD Catalyst Control Center didn’t appear to install correctly, but the computer now recognized the second monitor!

I often say that technology isn’t magic, but it seemed like magic!

So I went back to “devmgmt.msc”, went back to the properties for the Display adapter, and clicked on the “Events” tab. It displayed a history of driver installations for the day, and I noticed in the morning that it had installed “c0296217.inf” about 4 times. Then it added “Driver Service added (BasicDisplay)” and “Device installed display.inf”, which would’ve been the “Microsoft Basic Display Adapter”. Then… the final few entries said “Driver service added amdkmdag”, “Driver service added amdkmdag”, “Driver service added (AMD External Events Utility)”, “Device installed (c8160540.inf)”.

It wasn’t magic after all. Rather, the installer had used a different driver the final time! I searched C:\AMD for “c8160540.inf” and sure enough I found it at C:\AMD\Support\13-1-legacy_vista_win7_win8_64_dd_ccc\Packages\Drivers\Display\W86A_INF\C8160540.

I tried searching for “c0296217.inf” but I couldn’t find it in the C:\AMD directory. My guess… is that when I was trying to install the AMD Catalyst Control Center… it must’ve been using drivers that were already existing on the system. Even though the drivers didn’t work… it tried the pre-existing bad ones before it used the good new one that it had packaged with itself. Now that’s 100% a guess, but it’s clear that the final driver that works was one that I had downloaded with the 13.1 installer.

I wish I knew more about the overall process, but in the end I got my desired result, and that’s what matters most at the moment. Hurray for getting Windows 10 Pro to work on that machine with dual monitors! Also… hurray for being able to use remote desktop connection with dual monitors!

Now to pack up the hardware and get it home, and… spend the next 3 hours doing everything that I was originally going to do in 7.5 hours today.

Little victories…