Tag Archives: Python

by

Sending HTTP requests with Python: “httplib” vs “requests”

Some time ago, I decided that I wanted to learn Python. I was already quite experienced with Perl and PHP, and I wanted to add the third P to my repertoire.

So I decided to start writing all my personal code at home in Python, and overall I have quite enjoyed this decision. The majority of my personal software is now written in Python and I could ramble about why I enjoy working with it. I decided to push myself some more though, so I started writing data parsers in Python and now I want to start working with web APIs using it as well. Which brings me to “httplib” vs “requests”…

I can see why one of the first lines on the “httplib” documentation page is: “The Requests package is recommended for a higher-level http client interface” in Python (https://docs.python.org/2/library/httplib.html).

I read the examples in the docs, changed the code to match my needs, and ran the code. However, the responses to my requests were always “500 Internal Server Error”. Now… I was pretty sure the remote server wasn’t having a problem. I tried a different tool I’d written in Node.js and it worked fine. I tried a few other tests, and they worked fine. There was something wrong with the “POST” that I was trying…

I was getting a bit frustrated, and wanted a quick win, so I used “pip” to install “requests” and “requests[security”. I typed out a few lines and sure enough… I received a response of “200”. Beautiful!

But… I’m an insatiable problem solver. I wanted to know WHY “httplib” wasn’t working for me. I also didn’t really want to add a non-core Python dependency.

So I looked at the example code again… compared the code that differed between “httplib” and “requests”, and realized… it had to be the data I was sending in the body of the request. Or rather… it had to be a missing header describing the content that I was sending!

Sure enough, when I added “Content-type”: “application/x-www-form-urlencoded” to the headers I was sending, I met with success!

I don’t think I’ve ever had to do that with any library I’ve ever used before but apparently you need to be that explicit when using “httplib”. “requests” must take care of that under the hood.

Anyway, I just thought that I’d share that struggle. If I had just read the documentation for “httplib” completely clearly, it would have been obvious. I feel rather foolish for making that mistake, but… we learn by making mistakes! And hopefully this post helps out someone else out there.

It can be difficult learning a new language, especially when you’re doing it for fun. I am completely confident coding in Perl, so it was tempting to do this project in Perl. I probably would have been finished by now, because I could’ve just typed like the wind. But it’s about the process.

I know Perl, because I have years of paid and unpaid experience with Perl. I’ve used it a million different ways; I’ve made mistakes and found solutions. The same could be said for PHP, Javascript, and Bash/shell, as I’ve had to use those for countless hours at my day job.

But I don’t use Python at my day job. That means the hours I can invest in it are significantly fewer. It means it’s more of a challenge, but I think it’s a worthwhile challenge. It stimulates my brain, it’s fun to learn new things, and who knows when I might end up on a new project or a new job where I need to know Python?

Anyway, we’re getting into ramble territory. While I’ve figured out my problem with “httplib”, I might stick with “requests” just because it’s a bit more convenient. Additional dependencies be damned!

by

Running a process as a different user…

I’ve been doing a lot with configuration management lately. At work, I’ve used Chef to set up virtual machines on Amazon Web Services, and I’ve used Ansible at home to create and manage the configuration of my laptop and desktop. Both of these tools let you execute commands as different users than the one that invoked the original script. There are lots of reasons to do this, and it’s a great feature.

At the moment, I’m writing some custom Perl scripts, and I find that I also need to execute some commands as a different user.

So, by inspecting Ansible’s source code, I found that it just uses “sudo”. I figured Chef’s source code was probably a lot larger, so I asked around on IRC, and some very nice people on the #chef channel on freenode showed me the following links:

Rather than using sudo, Chef directly changes the EUID and UID of the process it’s running using the Ruby “Process” module.

A few nights ago, before I learned about how Chef handles it, I was reading about how to do this in Python and found “setuid” and “seteuid” with the Python “os” module.

In Perl, there are lots of ways to do it! If we look at http://perldoc.perl.org/perlvar.html, we’ll see that we can use “$<” and “$>” to change the UID and EUID respectively. That’s not very pretty though… so if you use the “English” module, you can refer to them as “$REAL_USER_ID” and “$EFFECTIVE_USER_ID” or “$UID” and “$EUID”. Or… you can use the “POSIX” module’s “setuid” method which will change both at once.

Cool, eh?

In my case, “sudo” would probably be fine, especially as it should take care of things like changing the group ID, the home directory, etc.

But let’s assume that we want to change the UID and EUID manually in our process. We won’t necessarily know what UID we want to use. We might just have a username. In that case, we can avail ourselves of Perl’s “getpwnam” function.

So let’s take a look at the documentation:

Well, that’s a bit sparse. Here are some alternatives:

So all we need is ‘my $uid = getpwnam(“postgres”);’
Then we can do something like ‘POSIX::setuid($uid);’ to change the UID and EUID of the current process!