Category Archives: Open Source

Virtualization with VMware vSphere (ESXi) Hypervisor

One of our servers, collocated in our office, crashed couple of weeks back. We took this opportunity to virtualize most of the stuff, earlier only some intances were virtualized.

Installing and setting up VMware vSphere Hypervisor is a straight forward process, even the process of creating virtual-machines.

Most of time, I spent was figuring out:

  • Storage devices and configuration
  • How many virtual machines, do we need?
  • Which operating systems, do we need?
  • Network configuration

I am not going to cover details of installation and network configuration. I might write more posts in future to cover specifics. Meanwhile, I would love to answer the comments, if I know or share anything I know.

Anyway, this is what we have now:

  • IBM x3250 M2
    • Intel Xeon (Quadcore) x64
    • 4 GB memory
    • 3TB+ storage connected to LSI RAID and SATA controller
  • Host: VMware vSphere Hypervisor – which runs out of  1.5 TB datastore, directly connected to SATA controller (different from LSI)
  • Guest VMs:
    • OpenVPN Access Server for VPN
    • Ubuntu 10.10 Server as Development sandbox
    • Zentyal(eBox) as Internet Gateway (filtering, IDs, proxy)
    • Ubuntu 10.04 LTS Server for SVN, Mails, Local Staging WebServer
    • Windows 2008 Server for ActiveDirectory, FileServer, Windows-Specific-Software (access-control-management-gui, etc.)
    • Others: Project Specific VMs

We are very happy with the improved performance and stability, and the way hardware is properly used. Thanks to VM templates, clones and snapshots, we would be able to provision above infrastructure with minimal downtime.

I would soon replace Windows Server with Zentyal (eBox), which I believe, can handle most of things. However, we would need another Windows XP box to run some software (proprietary), which only runs on windows, to manage various other devices (EPBX, Access-Control-System, etc.).

We now have relatively more scalable infrastructure, more virtual machines can be created to take care of email-server, svn-server, etc.

We are yet to invest in a sophisticated storage device such as DAS or NAS or SAN, hence we are relying on some cheap tricks here:

  • 1.5 TB disk as datastore for VMware vSphere, which stores
    • Guest Operating System VMs
    • Guest Operating System VMs Paging files
    • Virtual Disks (non critical data – programs and configs) required by host virtual-machines
  • 1.5 TB disk via Raw Device Mapping (RDM) shared among virtual machines
  • 500 GB disk via Raw Device Mapping (RDM) shared among virtual machines
  • 1.5 TB USB/Firewire – Backup storage
    • Rsync is used to backup critical data and configs from all virtual machines on daily, weekly and monthly basis.

Things I didn’t like:

  • VMware vSphere client requires Microsoft Windows
  • VMware vSphere client doesn’t allow creating RDMs from GUI, one has to use these instructions

Finally, I could execute some of things I planned more than two years back.

CalDAV and SyncML Resources

One of our projects requires a calendar application with bi-directional synchronization with any other calendar applications/services such as Microsoft Outlook, Mozilla Thunderbird, Google Calendar, Yahoo! Calendar, iPhone, Nokia or others.

We can export iCalendar file, that would be readonly i.e. client applications can not write back to source calendar

I started looking at CalDAV, which is an extension to WebDAV, and SyncML, which is gaining traction because most of the mobile devices use it for synchronization of data. I yet to find where CalDAV fits when it comes to mobile devices.

I found out some interesting links, which I am still going through. I thought, it would be nice to share these links:

Agilo: Scrum plugin for Trac

We use Trac as project collaboration space[1] within our company. We have been trying out agile-methodology (specially Scrum) for projects, but recently we went more formal with the approach using a software to manage the whole process.

I evaluated some software for agile project management and found good ones including ThoughtWorksMingle, but it’s not open-source and I believe, we can’t afford it (licensing, subscription).

We found ‘Agilo‘, a plugin for Trac, is easy to setup and was intuitive enough to get us started, considering our (no) experience with formal agile-methodology.

Do you want to get started with Scrum and don’t want to invest time learning a complicated software? You don’t need a software to do Scrum. However, if you like managing projects using a software and you are comfortable with Trac, try Agilo.

You might want to check Scrum in five minutes [PDF]

BTW! This is yet another post to spread the word about the software and tools, we find useful and have been using.

[1] project management, bug tracking, documentation, wiki and version-control browser

DeltaCopy: Backup software utility for windows

We have one Windows server, which is there for many reasons. Taking backup on windows can be pain, one has to rely on third-party software. We use DeltaCopy, which relies on rsync (an awesome piece of open-source software), statically linked Cygwin libraries/binaries.

It is easy to setup and doesn’t conflict with any existing Cygwin installation.

DeltaCopy has saved us from many disasters(disk failure, etc.), some of those happened during last one month. Had we not taken daily, weekly and monthly backup snapshots, we would have lost months of work.

Thanks to DeltaCopy and developers/company/community behind it. This post is my way of thanking them, by spreading some words.

Using DeltaCopy is very intuitive, however, feel free to leave comments, if you need any help setting it up.

Ideas: FlashDevelop on OSX and GNU/Linux

FlashDevelop is an open-source IDE for Flash Platform development (ActionScript, MXML, HAXE, etc.) on Windows platform. I have extensively used it in past, and still, painfully, use it via windows virtual-machine on OSX host.

Couple of years back, I looked at FlashDevelop code to figure out, if we can port it for other operating-systems, such as Mac OSX and GNU/Linux.

I posted on Flash Develop forum about my experiments:

I have been trying to run FlashDevelop under OSX. In last couple of years, whenever I got sometime, I tried these:-

  1. Run FlashDeveloper with DotGNU: FlashDevelop was running but due to a lot of native (Win32) API calls, lots of components were failing (screenshot: http://pic.im/5Xk)
  2. Run Flash Developer with Mono: I tried to compile FlashDevelop using MonoDevelop and run using Mono for OSX. Most of the problems is one component/library i.e. DockPanelSuite (http://sourceforge.net/projects/dockpanelsuite/), an opensource code that uses a lot of PInvoke calls (Win32 API).

I am really excited about the possibilities, if we can port this one library we can get very closer to run FlashDevelop on OSX using Mono.

Note: pics.im seems to be dead, so I have uploaded new screen-shots
Flash Develop on OSX - ScreenshotFlash Develop on OSX

Some Ideas:

MonoDevelop is good enough and works well on OSX, I guess also GNU/Linux. However, I am yet to look at licensing and possibilities of forking the project (MonoDevelop)?

BTW! I have been lazy to write about such experiments on the blog, but while commenting on Jesse Warden’s post, I thought, it makes sense to post it here, so it lives.

Please let me know, if you have any ideas about making this happen? I would be happy to collaborate, contribute and donate in whatever capacity I can.

Eclipse Plugins

Some Eclipse plugins, I use (with Eclipse Classic SDK), are:

There are bunch of plugins for network monitoring and debugging, don’t remember their names. BTW! I have got two separate installations (32bit Carbon and 64bit Cocoa) of Eclipse. I hope, Adobe would port Flex/Flash Builder to work with 64bit and Cocoa.

Beautiful Plogger

Plogger is an open-source photo gallery system built using PHP. It successfully maintains the simplicity while providing so many features, it’s just beautiful piece of work.

Withing five minutes, I got Plogger running on my machine. In another hour, I created a new theme with different visualization (CoverFlow) to display thumbnails.

Having worked many different photo gallery management software/scripts, I find, Plogger is the most beautiful. I might be ignorant about other better systems, please share if you know.

I bet, web designers would find it very handy because they can easily customize and deliver solutions without dealing with complex programming required behind file upload/management.

Open Source Media Framework (OSMF)

A few days back, Adobe announced Open Source Media Framework (OSMF), which was known as Strobe earlier.

A few minutes back, I received an email confirming that Open Source Media Framework website is up with all the information, specification, developer-documentation, source (Subversion repository), downloads (binary and source) and bug-database.

Open Source Media Framework (OSMF) has an interesting list of features and goals. I like it’s plug-in architecture, that would allow a lot of extensibility and various amazing plug-ins from different providers in coming time.

If you are a developer spending a lot of time developing various work-flows around online media-players (audio, video, images – players), I strongly recommend you to check out Open Source Media Framework (OSMF). I am sure, you would find it worth start using it and also contribute back whatever you can.

Following are the reasons, I am going to use OSMF and give up my existing frameworks:

  • It’s open-source
  • It would solve core issues (video playback, delivery and workarounds for various limitations in Flash Player, etc)
  • It would adopt various standards (playback, cdn, advertising, etc)
  • Larger developer base, that means bug fixes and various features would be available very often
  • I love open-source and see myself contributing to OSMF in future, beyond my current projects.

OpenFiler – an opensource NAS

We wanted to have a scalable storage system. So I went ahead and checked out the cost of buying NAS from various vendors (NetApp, Dell, etc). I figured out, it was going out our of budget. Then I started learning, what it takes to build a NAS for a small teams like ours.

I started evaluating various FOSS NAS options for our office. I checked out FreeNAS and OpenFiler, finally decided on OpenFiler.
I chose OpenFiler for simple reasons – stability and production-quality. Whereas, FreeNAS has a lot more features, than OpenFiler, but doesn’t look that stable. Perhaps, in future I might go for FreeNAS for it’s various cool features (UPNP, iTunes streaming, etc).

We are using an old server based on Intel’s Server Entry Board, Pentium 4 processor, one Gigabyte memory, one IDE drive and two SATA drives. Both SATA drives are under RAID 1 configuration using OpenFiler’s software RAID. I am planning to get RAID controller card so we can use more disks.
OpenFiler boots from USB flash-drive, to make this happen it took some extra effort, Thanks to [email protected]. FreeNAS provides images for flash-devices, so it’s lot easier to boot FreeNAS from USB flash-device. BTW! Booting OpenFiler or FreeNAS from USB flash-drive would save one IDE/SATA port on motherboard, which can be used to plug-in another harddrive for better purpose (not for booting small NAS OS).

Anyway, we are using Intel NAS Performance Toolkit to benchmark our NAS server. We are also doing a lot of tests (semi)manually. The idea is to cover all cases and also come up with disaster recovery strategy.
I would post more details on our findings/benchmark-tests, so it helps you, if you plan to go for it.

Technorati tags: , ,

Development Setup

I have been thinking a lot about a good and scalable development setup for small teams.That means, as team size grows, things don’t have to be changed.

Server machine(s):-

I am planning to have all of these one server machine (IBM) in virtualized environment. Subversion and Trac can go in one VM, perhaps under Apache. Second VM can have Apache for web-apps, third VM can have mysql, fourth VM can have MailServer and ActiveDirectory/LDAP?

Do you think above makes sense?

Developer/Designer workstations:-

Though there should not be any restrictions on using a development IDE of choice, but I have realized if entire team uses same IDE things get far easier. The development workflow is easy, knowledge sharing and management is easy. Eclipse, which Aptana is based on, can be used for variety of projects.

Ubuntu is good operating system, I have been using for over one year, not regularly but quite often. There might be Windows/OSX boxes for designers.

Why do we need Apache and MySQL on each workstation?

I think, developers are more effective if they have their own sandbox. They can play with code freely, take certain risks and innovate. There is no harm done. We do face issues, when it comes to sync mysql data into local mysql instances, I think it can be done via some scripts.

Virtualization can be done on client and I am thinking to do it, specially for folks who are not comfortable with Linux, so they can be on Windows while they also have access to Linux VM, they are encouraged to use Linux VM. They can use Windows box (Host), if something is messed up in Linux VM and there is strict timeline that doesn’t allow to fix VM.

Over the period, these developers would be comfortable with Linux and that is the time they can start using real Linux box 🙂

These are some thoughts, I might be missing something here. I hope, I would write more in another post.

As you know, I have been on break and been thinking on all these things. BTW! I don’t work with Yahoo! anymore, once I am done with break I have to figure out what to do. Perhaps, time to be in my own city and do something my own (consultancy, startup or something like that). Heh, that’s news, if you have read so far 🙂

Technorati tags: , , , , , , , , , , , , , ,