Creating and Customising Amazon EC2 AMI

Do you want to create/customize an EC2 AMI?

Check out this tutorial by Rohan.

Briefly, steps are:

  • Find a suitable base AMI. You can use EC2 Console to search
  • Launch an instance from command-line using ami-id or EC2 Console
  • Copy(scp) your private (ssh) key, ec2 key-pair and certificate from your machine to instance’s /mnt directory
  • SSH to instance
    • Install/customize tools and configuration
    • Test if things work
    • Install amazon-ami-tools
    • Bundle image (from current running instance):
      ec2-bundle-vol -d /mnt -k /mnt/pk-STRINGOFTHIRTYTWOLETTERSORDIGITS.pem -c /mnt/cert-STRINGOFTHIRTYTWOLETTERSORDIGITS.pem -u XXXXXXXXXXXX

      XXXXXXXXXXXX is your Amazon account number without dashes.

    • Deploy/copy AMI files to S3:
      ec2-upload-bundle -b [your-s3-bucket] -m /mnt/image.manifest.xml -a [aws-access-key-id] -s [aws-secret-access-key]

Before bundling the image, do test customised settings and configuration; it’s pain to repeat above steps again, in order to change anything post-deployment.

Tip: Don’t shutdown/terminate the instance, you used to create AMI, until you have tested instances (based on new AMI). If anything goes wrong with new instances, you can come back to this instance (based on base-ami), change, bundle and redeploy.

Amazon S3 RequestTimeTooSkewed Error

Recently, we have started noticing “RequestTimeTooSkewed” erro while making requests to Amazon S3. The official FAQs suggests following:

Amazon S3 requires all machines making requests be within 15 minutes of an Amazon S3 webserver’s clock

Reading documentation and various blog posts made me believe, this is problem with the date/time settings on machine (which makes request). However, in our case, it turned out to be something else.

We use boto (Python library) for AWS stuff. Boto caches the connection objects (for S3, EC2, SQS, etc.), and tries to reuse those in later calls.

Imagine a simple use-case:

  1. Read data from S3
  2. Process data on EC2-instance/your-server
  3. Store processed-data on S3

In our case, #2 takes longer than 15 minutes (allowed limit by S3) sometimes, hence,  we were noticing ‘RequestTimeTooSkewed’ error.

Anyway, we fixed it by explicitly creating a different/fresh S3(http) connection before sending request to S3. This might sound inefficient, perhaps it is.

This is how we do now, a stripped down version of code from our class:

s3_connection = boto.connect_s3(aws_access_key_id="aws_access_key_id",aws_secret_access_key="aws_secret_access_key")
output_bucket = s3_connection.get_bucket("output_bucket")
key = output_bucket.new_key(key_name)
key.set_contents_from_filename(file_path, file_headers, True, None, 10, file_policy)

I am Python and boto noob; I am sure there would be a better way of doing this, so please share if you know one.

Thoughts: Open Standards and Technology for Movie Rental Business

I have been thinking about open-standards for movie-rental business. I searched to find about such standards but I couldn’t locate. Do you know of any?

Basically, a set of specifications/protocols which can be implemented by different players in movie-rental chain. Players include studios, distributor, application-developers, portals, et. al.

Rough thoughts:

  • A studio/distributor implements this specification/protocol and exposes API  for metadata/streaming(or playback)/file-access from their own server
  • Third party developers use the API to to pull metadata and play the content
  • Third party developers can charge the user and split the revenue with content owner
  • Or, studios/distributors can enable payment-gatway APIs as part of their offering
  • Content is always played directly from studio/distributor’s server
  • Studio/distributor have total control over their assets and also charge based on subscription/per-access
This would solve many problems (billing, stats, etc.), however, there would be concerns:
  • someone (end-user, third-party developer) might try to cache/save streams as file and distribute
  • and other concerns?
Once we have specifications and protocols in place – a stack using open-source technologies can be built as template, and made available so that everyone in chain can use/customize it to build their part.
I wanted to talk about it on twitter but 140 chars were too short to express my thoughts. Anyway, it’s good to have a blog post in place.
Please share your views.

Tata Photon+

I got Tata Photon+ mobile broadband connection during 2010, it worked well initially for a month or two.I can notice, their services have been degrading since then (late 2010).

Connectivity is not an issue, but download speed and latency.

A recent test results done on June 6, 2011
Tata Photon+ Speed Test 1 against Delhi Server Tata Photon+ Speed Test Results - Mumbai Server

We all know, most of marketing/advertisements campaigns are big lies.  There is big disconnect between the idea/dream they show and the reality.

In the case of Tata Photon+, they advertise about giving ‘up to 3.1 mbps’ download speed BUT I am yet to get 1 mbps consistently for an hour, leave alone days.

I guess, this is location specific. Places like Kanpur and other cities in UPE are not given enough of importance (network infrastructure and attention)?

Whenever I complain to Tata Photon Customer Support, someone visits, tests using speedtest.net against Delhi/Mumbai servers, and shows me test results (skewed?) which is generally in range of 600 kbps to 1.2 mbps.

When I show him results of tests done against servers in USA and Europe, they would argue  - which reflects their ignorance about technology and terminology (ping, latency, etc.)

Yesterday, I filed another complaint over phone. I got a call asking if I was in front of computer, I said “No, but I would be in couple of hours and tomorrow”. After couple of hours, I got a SMS suggesting my complaint-ticket had been resolved and marked as closed. Really? No one visited me.

I filed another complaint using web-site, got around 10+ calls within 12 hours. All of them asking same question “What’s problem sir”. I don’t understand, why didn’t these guys look at their CRM and read the details (which included the speedtest.net test results)?

Anyway, one of those guys (from some Tata hired agency) visited me today. He repeated the same process of testing against two Indian servers using speedtest.net. This time, he got lucky speed results showed around 1 mbps agaisnt Delhi/Mumbai server. I requested him to visit again tomorrow in afternoon and perhaps at different location.

During middle of meeting, this fellow stood up and walked out. I followed him to ask what happened?

He misbehaved by saying “I heard enough, I am going. That’s all we can do”.

I felt offended, I asked about his details, he denied giving by saying “Do whatever you can”. He left ignoring me.

I feel bad as customer, and also because I have been a Tata employee. I had never expected, one of Tata companies would do such things i.e. poor service and worst customer support (not at phone, but at ground level – those visiting agents).

I feel, I have wasted so many hours talking, thinking and complaining. No one wants to listen and do anything.

Updates:

  • June 24, 2011: I found Tata Photon very slow at my residence, hence filed a fresh complaint (requested that someone should visit me on Sunday at my residence to test speed in that area) via phone. CSR said, they don’t work on Sunday, I said – I can be available on weekdays provided an appointment is taken.
  • June 25, 2011: I received two calls from Tata Photon support team. In both calls, I was asked same question “what is problem, sir?” with one difference, in latter call I was told that they can visit me on Sunday
  • June 26, 2011: I got a call, where I was asked same  and some more questions:
    • What is problem, Sir?
    • Did you buy device with warranty?
    • What speed do you get?

    I told them, “I am happy with the device but not with the signal and speed in this area (my residence), hence needed someone to visit me and test speed here”. They asked me to go to Tata Photon centre, around 8 kms from my place. I asked, “how can they test issues of this area in different area?”.

    They threw another bomb by saying, a fee( INR 75) needs to be paid by customer, if he/she wants technical-support-engineer to visit customer’s place?

    I got upset and reminded that this is mobile-broadband. Isn’t it their responsibility to make sure connection signal and quality is good in all areas?

    Anyway, that was new, I had not paid such fee earlier.

    I asked, when did they start charging? She said, “Yesterday”. I reminded her, I filed complaint day before yesterday. She said, it has been started on June 20, 2011.

    I asked her to pass the call to her senior, who ended up saying the same thing. I told them, the policy is ridiculous.

Why a customer has to pay extra to get service related issues checked/fixed? Isn’t customer already paying every month? Isn’t it Tata’s responsibility to provide services as they promise?

I have invested some money and time to get the device and connection; I can switch to different provider, but device might not be reusable unless someone buys it from me.

Once again, I am disappointed by Tata Tele’s services. I am going to move away from them and would not recommend Tata Tele’s services to my family and friends.

More Updates:

  • June 28, 2011: As an last attempt, I wrote an email to Tata Tele Services contact and also copied the same to UP East Nodal Officer.
  • June 29, 2011: I got a call from a senior Engineer, who visited me on the same day. During 3-4 hours long troubleshooting, he figured out the reason behind slow speed, which is, HSIA signals was not as strong as CDMA 1X, and some tower issues.He got an antenna installed, on the roof-top, and connected to my USB device. He promised to get a repeater installed during weekend, so that all devices at our place can get better quality signals.
    Once antenna was installed, and some augmentation was done in one of the towers, things improved a bit.

    He seemed a good professional and knew the stuff. He earned my respect. I learnt, good guys are very overloaded because someo of their peers don’t do their work properly :-)

  • July 2, 2011: Two Tata Photon smart junior-engineers visited and installed a repeater at our house. The senior engineer kept his word, he earned more respect.I can feel the difference now; Two different tests during last 5 hours have given very positive results, as shown below:

    Tata Photon Speed Test Tata Photon+ Speed Test Result

    Tata Photon+ Speed Test using 2Wire.com

    Once again, I gave my feedback about service and the misbehaviour by one of their peers. I am sure, the good-guys would fix things by putting peer pressure and leading by examples.

    As far as fee for visit is concerned, I was told by one of the engineers, it is for the cases where customer’s device or computer is found to be faulty, not the service. I wonder, why I was not told the same earlier during the call, which made me upset?

    I am going to wait for couple of days. If I get consistently better quality speed, I would strike out the line (about not recommending) above. Anyways, there are some issues with Tata Photon+ customer support and service, which would need to fix, so that all issues can be resolved without any escalation.

    Tip: Choose HSIA from network-settings of Tata Photon+ dialer, this would force modem to use to HSIA enabled towers/frequencies and would avoid using CDMA 1X.

Internship Opportunities @ Allies

We have been working on some exciting stuff for over last three years, we want to take that to next  level. We also have some crazy and cool ideas for new products/saas, we have got enough details figured out to get started.

We want some smart interns to work with us during this summer or longer (perhaps six months or full-time :-) ) in any or all of following:

  • Web Development (goal: build new products or extend existing products)
  • Mobile (goal: turn our ideas into reality by building some cools apps and games for iPhone, Android and other mobile-devices)
  • Flash Platform (goal: build some cool applications for our existing project)

Summer vacation has started in most of the colleges in India; Some passionate students would want to do something more useful during this time.  If you are one of them, why don’t you get in touch with us?

BTW! We are always looking to hire smart and passionate designers and developers for our team. If you are interested into a full-time position, please let me know.

Development Setup – Updated

I posted about our proposed development setup, during these years a few things have changed a bit. More or less, this is what we have:

  • GNU/Linux (Ubuntu) and Windows(some) boxes for developers
  • Eclipse (PDT) IDE with loads some plugins
  • Version Control: Subversion, GIT, GIT-SVN
  • Browsers
    • Firefox
    • Chrome
    • Opera
    • Safari
    • OperaMini (Simulator)
    • FirefoxMobile (Simulator)
  • Plugins for browsers:
    • Firebug (with loads of plugins – YSlow, etc.)
    • Web Developers Tools (firefox)
    • Others (developer’s preference)
  • Developers also use WINE to run some windows software on GNU/Linux

Testing:

  • Windows Box with different browsers (versions, vendors)
  • Windows Box with different Mobile Emulators/Simulators
  • Selenium Plugin for Firefox

Development Servers/Applications:

  • Apache HTTPD
  • Mysql Server
  • Trac
  • MediaWiki
  • PHPMyAdmin
  • ReviewBoard (limited usage)
  • Hudson (limited usage)

Virtualization with VMware vSphere (ESXi) Hypervisor

One of our servers, collocated in our office, crashed couple of weeks back. We took this opportunity to virtualize most of the stuff, earlier only some intances were virtualized.

Installing and setting up VMware vSphere Hypervisor is a straight forward process, even the process of creating virtual-machines.

Most of time, I spent was figuring out:

  • Storage devices and configuration
  • How many virtual machines, do we need?
  • Which operating systems, do we need?
  • Network configuration

I am not going to cover details of installation and network configuration. I might write more posts in future to cover specifics. Meanwhile, I would love to answer the comments, if I know or share anything I know.

Anyway, this is what we have now:

  • IBM x3250 M2
    • Intel Xeon (Quadcore) x64
    • 4 GB memory
    • 3TB+ storage connected to LSI RAID and SATA controller
  • Host: VMware vSphere Hypervisor – which runs out of  1.5 TB datastore, directly connected to SATA controller (different from LSI)
  • Guest VMs:
    • OpenVPN Access Server for VPN
    • Ubuntu 10.10 Server as Development sandbox
    • Zentyal(eBox) as Internet Gateway (filtering, IDs, proxy)
    • Ubuntu 10.04 LTS Server for SVN, Mails, Local Staging WebServer
    • Windows 2008 Server for ActiveDirectory, FileServer, Windows-Specific-Software (access-control-management-gui, etc.)
    • Others: Project Specific VMs

We are very happy with the improved performance and stability, and the way hardware is properly used. Thanks to VM templates, clones and snapshots, we would be able to provision above infrastructure with minimal downtime.

I would soon replace Windows Server with Zentyal (eBox), which I believe, can handle most of things. However, we would need another Windows XP box to run some software (proprietary), which only runs on windows, to manage various other devices (EPBX, Access-Control-System, etc.).

We now have relatively more scalable infrastructure, more virtual machines can be created to take care of email-server, svn-server, etc.

We are yet to invest in a sophisticated storage device such as DAS or NAS or SAN, hence we are relying on some cheap tricks here:

  • 1.5 TB disk as datastore for VMware vSphere, which stores
    • Guest Operating System VMs
    • Guest Operating System VMs Paging files
    • Virtual Disks (non critical data – programs and configs) required by host virtual-machines
  • 1.5 TB disk via Raw Device Mapping (RDM) shared among virtual machines
  • 500 GB disk via Raw Device Mapping (RDM) shared among virtual machines
  • 1.5 TB USB/Firewire – Backup storage
    • Rsync is used to backup critical data and configs from all virtual machines on daily, weekly and monthly basis.

Things I didn’t like:

  • VMware vSphere client requires Microsoft Windows
  • VMware vSphere client doesn’t allow creating RDMs from GUI, one has to use these instructions

Finally, I could execute some of things I planned more than two years back.

TEDx in Kanpur – TEDxIITKanpur

Entrepreneurship Cell (E-Cell), at IIT Kanpur, is organising TEDxIITKanpur event on March 12, 2011. This is the first event of it’s kind, and I hope we would have more of those (TEDx, BarCamp, DevCamp, etc.) in Kanpur (within IITK campus or outside).

Our company (Allies Interactive Services) is honoured to contribute as design sponsor of TEDxIITKanpur, and hopefully other events organized by E-Cell, IIT Kanpur.

Adobe AIR and Flash Player Incubator program

The Adobe Flash Platform runtimes team has launched Adobe AIR and Flash Player Incubator program, which would showcase bleeding-edge, under-development or under-consideration features of Adobe AIR and Flash Player builds.

Unlike Adobe pre-release program(s), Incubator program is open to everyone, and encourages companies and developers to look at under development features, and also give feedback to shape the future of these runtimes.

This launch is another major milestone, after Adobe Labs and Open Source initiatives by Adobe. We are heading in right direction.

“Molehill” 3D APIs and Cubic Bezier Curves are the first two features available in incubator build. I am sure, there would be more features available in coming months.

BTW! These early features might not make to final release of runtimes, as it’s always case with

Visit Adobe AIR and Flash Player Incubator’s page on Adobe Labs to learn more. You can also subscribe to new Adobe Flash Platform Runtimes releases blog to get real-time updates on new AIR and Flash Player releases.

A Developer’s Open Letter to RIM

Jamie Murai has written an open letter to RIM, the company behind Blackberry and Playbook. Jamie talks about his experience of setting up development environment for Playbook application development.

It seems, RIM has failed to attract developers, the credit goes to broken development process and tool-chain. One has to spend good amount of time (and money?) setting up development environment, building application and deploying the same to marketplaces.

Why would developers bother looking at Playbook development, when they have better options (iOS and Google Android), where  development, deployment and monetization is relatively very-very simpler.

Developer usability is very important. Any company that relies on developer community/ecosystem for their growth, should take some lessons from Jamie Murai’s open-letter, which enlists some important points/issues.

Read Jamie Murai’s Open Letter To RIM’s Developer Relations.

CalDAV and SyncML Resources

One of our projects requires a calendar application with bi-directional synchronization with any other calendar applications/services such as Microsoft Outlook, Mozilla Thunderbird, Google Calendar, Yahoo! Calendar, iPhone, Nokia or others.

We can export iCalendar file, that would be readonly i.e. client applications can not write back to source calendar

I started looking at CalDAV, which is an extension to WebDAV, and SyncML, which is gaining traction because most of the mobile devices use it for synchronization of data. I yet to find where CalDAV fits when it comes to mobile devices.

I found out some interesting links, which I am still going through. I thought, it would be nice to share these links:

Adobe Flash Platform and I

It appears to me (and perhaps others), I have distanced myself from Adobe Flash Platform during last couple of years. Not because I don’t like it, actually I love it, rather due to some other priorities here at our company, which does lot more (web-standards and design) than flash/flex stuff.

All this time, I have been keeping very low profile in flash-platform community, literally not talking about anything. I feel bad about it, I don’t know ‘why’.

I have done some trivial stuff (video players, flashlite and air apps) but nothing very cool or useful to everyone else.

My to-do list contains a long-list of applications targeted towards Adobe Flash Platform (desktop and other devices), I am supposed to do; Considering, my plate is full with all other responsibilities, I am not spending even 1/4th of the time writing actionscript.

Meanwhile, I have tried to encourage my team to learn actionscript and flex-framework. I even tried to hire some flash (or flex?) developers. I even managed to hire one flex developer last year.

Unfortunately, this person left the company as soon as when it was time to contribute back after 6 months training (no contribution period). That was big set back, a discouragement to me; I stopped looking for Flash/Flex developers.

I keep on promising myself, that I would do these applications; I keep on forgiving myself when I am not getting those done.

Anyway, there is still hope, I would building applications the on language (ActionScript), I am very good at. There is still hope, this blog would have more posts related to Adobe Flash Platform.

Agilo: Scrum plugin for Trac

We use Trac as project collaboration space[1] within our company. We have been trying out agile-methodology (specially Scrum) for projects, but recently we went more formal with the approach using a software to manage the whole process.

I evaluated some software for agile project management and found good ones including ThoughtWorksMingle, but it’s not open-source and I believe, we can’t afford it (licensing, subscription).

We found ‘Agilo‘, a plugin for Trac, is easy to setup and was intuitive enough to get us started, considering our (no) experience with formal agile-methodology.

Do you want to get started with Scrum and don’t want to invest time learning a complicated software? You don’t need a software to do Scrum. However, if you like managing projects using a software and you are comfortable with Trac, try Agilo.

You might want to check Scrum in five minutes [PDF]

BTW! This is yet another post to spread the word about the software and tools, we find useful and have been using.

[1] project management, bug tracking, documentation, wiki and version-control browser

DeltaCopy: Backup software utility for windows

We have one Windows server, which is there for many reasons. Taking backup on windows can be pain, one has to rely on third-party software. We use DeltaCopy, which relies on rsync (an awesome piece of open-source software), statically linked Cygwin libraries/binaries.

It is easy to setup and doesn’t conflict with any existing Cygwin installation.

DeltaCopy has saved us from many disasters(disk failure, etc.), some of those happened during last one month. Had we not taken daily, weekly and monthly backup snapshots, we would have lost months of work.

Thanks to DeltaCopy and developers/company/community behind it. This post is my way of thanking them, by spreading some words.

Using DeltaCopy is very intuitive, however, feel free to leave comments, if you need any help setting it up.

Hiring Mistakes

Hiring mistakes hurt us[1] in terms of time, money, cultural-issues and many other things. I have made those, more than one time – that makes me guilty of repeating mistakes (a crime or sin?).

We filtered résumé, screened candidates over phone, interviewed in-person over multiple rounds and discussed among ourselves before extending employment offer. Sounds good to get good candidates? No, it is not. We ended up having some bad hires.

Anyways, most of us know – Resume don't matter anymore; Someone really good at things[2] should have more than résumé to show and talk about

  • bunch of projects done in free time;
    • indicator: passion, love for craft and desire to improve by practicing in free time
  • technical or craft related blog
    • indicator: communication skills; knowledge sharing attitude
  • contribution to open-source projects
    • indicator: love for FOSS; desire to do something beyond self, for others; team player
  • active participation in technical communities[3]
    • indicator: knowledge sharing attitude; mentoring and helping
  • every word on résumé should be justified
    • indicator: modesty and humbleness

I believe, we lacked some discipline in our hiring process and new-hire-orientation process. We were also not strict, at reviewing and monitoring progress, during probation period.

Both pre and post hiring phases are important.

Pre hiring is good time to know about candidate personality wise.

Post hiring is another chance to make sure if hiring decision was really good. Within three months, it should be absolutely clear (binary: 0 or 1), whether one is good match[4]

The one who has potential to be rockstar and great team-player should be given more time and mentoring; This might affect the economy of company in short term, but worth doing for long term benefits. Patience works, I need to get better at this.

I am learning something everyday, often I realize I am not doing something right; I need to bring discipline in my life and the way I do things, hopefully, rest would go very well.

In Kanpur, I am yet to connect to good software developers and designers. I know there are good ones, perhaps they are working outside of Kanpur.

I am curious, how do you hire, train and get people started?

[1] Entrepreneurs and companies
[2] Programming, Software development, Graphic/UI/UX Designing or skill in question
[3] flex_india or alike), forums, etc.
[4] Great Thinker, Technically sound and has emotional-intelligence