Category Archives: Python

Installing NumPy and SciPy on Mac OSX 10.8 (Mountain Lion)

I am playing with ExpEyes, which is awesome tool for anyone who wants to learn electronics (and physics).

ExpEyes comes with software (written in python) for GNU/Linux and Windows. I intend to run it on Mac OSX, because it can (provided all dependencies are met).

It is generally easy to install python packages on OSX using easy_install or pip, however, installing NumPy and SciPy turned out to be pain on OSX (10.8.2, latest version the time of posting).

In my case, I am using python binary installed via homebrew, so I can’t take advantage of numpy (which comes installed with mountain-lion).

I have XCode 6 installed, which comes with gcc-4.2.1 (llvm build). I figured out, I can build numpy and scipy from source.

Following is what I did to build numpy and scipy. SciPy requires gfortran, so that has to be installed (I used homebrew to do that):

$ git clone https://github.com/numpy/numpy.git
$ git clone https://github.com/scipy/scipy.git
$ cd numpy
$ python setup.py build && python setup.py install
$ brew install gfortran
$ cd ../scipy
$ python setup.py build && python setup.py install

BTW! In case, you are wondering, why didn’t I use Scipy Superpack script, which indeed makes job easier?

I didn’t, because I want to install gnu/unix packages using homebrew, and I already have git and other things. Super Spicypack downloads (gfortran, etc.), builds and installs packages directly to system, making it harder to remove those later? Homebrew makes job easier.

RIP, Kenneth Gonsalves “KG” (1953 — 2012)

Kenneth Gonsalves(Image Courtesy: Linux Bangalore (foss.in))

Kenneth Gonsalves, a champion in Indian (NRC)FOSS, Python and Django community passed away on August 3, 2012. Kenneth’s presence would be hugely missed in the community.

In 2008, Tarun Dua introduced me to Kenneth. I interacted with him at various occasions about NRCFOSS and general stuff. I was very inspired to see his contributions, specially some of his open-source projects.

While checking out NRCFOSS website today, I came to know about this sad news.

May his soul rest in peace.

Amazon S3 RequestTimeTooSkewed Error

Recently, we have started noticing “RequestTimeTooSkewed” erro while making requests to Amazon S3. The official FAQs suggests following:

Amazon S3 requires all machines making requests be within 15 minutes of an Amazon S3 webserver’s clock

Reading documentation and various blog posts made me believe, this is problem with the date/time settings on machine (which makes request). However, in our case, it turned out to be something else.

We use boto (Python library) for AWS stuff. Boto caches the connection objects (for S3, EC2, SQS, etc.), and tries to reuse those in later calls.

Imagine a simple use-case:

  1. Read data from S3
  2. Process data on EC2-instance/your-server
  3. Store processed-data on S3

In our case, #2 takes longer than 15 minutes (allowed limit by S3) sometimes, hence,  we were noticing ‘RequestTimeTooSkewed’ error.

Anyway, we fixed it by explicitly creating a different/fresh S3(http) connection before sending request to S3. This might sound inefficient, perhaps it is.

This is how we do now, a stripped down version of code from our class:

s3_connection = boto.connect_s3(aws_access_key_id="aws_access_key_id",aws_secret_access_key="aws_secret_access_key")
output_bucket = s3_connection.get_bucket("output_bucket")
key = output_bucket.new_key(key_name)
key.set_contents_from_filename(file_path, file_headers, True, None, 10, file_policy)

I am Python and boto noob; I am sure there would be a better way of doing this, so please share if you know one.