Category Archives: AWS

Docker Global Mentor Week 2016 – Kanpur

Docker Mentor Week 2016 - Kanpur

I have not been blogging quite often but you must have guessed from my posts that I love Docker. It’s beautiful piece of technology which comes with a lot of great tooling for developers, devops and infrastructure administrators.

I am Docker Kanpur Meetup Group organiser. We are organising Docker Global Mentor Week in Kanpur on November 18, 2016.

If you are from Kanpur or around (Lucknow, Unnao, etc.), please do join the meetup group, and participate in this and future events. It’s lot of learning, and fun to be at meetups.

For your convenience, I am embedding event information below even though it’s available on meetup page. RSVP is must, and we only have space for 20 people. Please confirm it on meetup page.

I look forward to meeting you at the event. Let’s make it fun and successful.


Join us for our Kanpur edition of the Docker Global Mentor Week!

Docker Global Mentor Week is your opportunity to #learndocker. Right now, Docker is working on self paced online labs that will be available through LearnDot. One lab will be for beginners and one lab will be for intermediate users. Docker mentors will be on hand at this event to help you as you work through the self paced materials.

Are you an advanced user? We strongly encourage Docker users of all skill levels to attend! We need a network of mentors who understand the Docker platform to answer any questions that newcomers may have. Click here to sign up as a mentor.

Agenda: November 18, 2016

  • 16:00 – Doors Open
  • 16:15 – Welcome message, presentation of Docker & snacks
  • 16:30 to 18:30 – Select and work through self-paced lab
  • 18:30 – Socializing

As we get closer to the event, we will share more information including materials on how to prepare for the training.

Pre-Event Set-up

At the training, you will need to bring your own computer. Before you go to a mentoring event, there are some steps you should do some preparation to get your work environment ready. This will help get ready for the tutorial, and also save on bandwidth at the event venue. Here are the steps:

Set-up Docker on your machine

For Linux users, we need you to install Docker engine and Docker compose. Make sure you have Docker compose version 1.6 or higher by running docker-compose version from the command prompt.

For Mac users, install Docker for Mac or if you have an older Mac, Docker Toolbox.

For Windows users, if you have Windows 10 pro install Docker for Windows, otherwise install Docker Toolbox.

If you are a Docker for Windows user and want to try the new Windows containers, you need to have fully updated your machine. To prepare to use Windows containers,

Go through the setup steps in the Windows Container lab.

In Powershell run
docker pull microsoft/windowsservercore:latest
This will take awhile, but it is essential you do this before going to the venue. The Windows base image is a 3 gig download.

If you’re new to Docker, pre-pull the docker images for the very basic tutorial
  • docker pull hello-world
  • docker pull alpine
  • docker pull seqvence/static-site
To run the application and participate in the rest of the training, pre-pull these images
  • docker pull microsoft/dotnet:1.0.0-preview1
  • docker pull node:5.11.0-slim
  • docker pull python:2.7-alpine
  • docker pull redis:alpine
  • docker pull postgres:9.4
For the orchestration part, you will want to pre-build the demo app by running the following steps
  • git clone git://github.com/jpetazzo/orchestration-workshop
  • cd orchestration-workshop/dockercoins
  • docker-compose build

Venue:

Allies Interactive
18/179, 3rd Floor
Above Karur Vyasya Bank,
Next to Hotel Geet or Akashdeep,
The Mall (Phoobagh), Kanpur-01
Phone: +91 512 2351352

Amazon Cloudfront – Video Streaming (rtmp) Issue?

Are you using Amazon Cloudfront for public/private video streaming over rtmp? Having trouble to figure out why some of your videos fail to stream (over rtmp) and throw ‘Stream not found’  flash/actionscript client side errors?

I have figured out:

  • Videos with invalid chapter markers would not stream over rtmp (Adobe Flash Media Server) or via Cloudfront.
  • Adobe’s FLVCheck utility suggests our videos have invalid chapter markers. I am not sure, if chapter markers in our videos are invalid or Adobe Media Server expect those to be in different format.
  • Same set of videos play fine with Quicktime/VLC player, Adobe Flash Player (over http progressive download) and html5 player (chrome or others which support H.264 videos)

I am disappointed with the fact – Adobe Media Server ceases to play (core feature) a video because of (optional feature?) chapter marker(s) which can be ignored if it is corrupt or invalid.

Thanks to AWS engineers who confirmed the case, and provided more details.

Update: Thanks to Justin Riley for sharing a solution (in comments below). He suggested to use ‘-map_chapters -1’ to get rid of this issue. It works for me.

Creating and Customising Amazon EC2 AMI

Do you want to create/customize an EC2 AMI?

Check out this tutorial by Rohan.

Briefly, steps are:

  • Find a suitable base AMI. You can use EC2 Console to search
  • Launch an instance from command-line using ami-id or EC2 Console
  • Copy(scp) your private (ssh) key, ec2 key-pair and certificate from your machine to instance’s /mnt directory
  • SSH to instance
    • Install/customize tools and configuration
    • Test if things work
    • Install amazon-ami-tools
    • Bundle image (from current running instance):
      ec2-bundle-vol -d /mnt -k /mnt/pk-STRINGOFTHIRTYTWOLETTERSORDIGITS.pem -c /mnt/cert-STRINGOFTHIRTYTWOLETTERSORDIGITS.pem -u XXXXXXXXXXXX

      XXXXXXXXXXXX is your Amazon account number without dashes.

    • Deploy/copy AMI files to S3:
      ec2-upload-bundle -b [your-s3-bucket] -m /mnt/image.manifest.xml -a [aws-access-key-id] -s [aws-secret-access-key]

Before bundling the image, do test customised settings and configuration; it’s pain to repeat above steps again, in order to change anything post-deployment.

Tip: Don’t shutdown/terminate the instance, you used to create AMI, until you have tested instances (based on new AMI). If anything goes wrong with new instances, you can come back to this instance (based on base-ami), change, bundle and redeploy.

Amazon S3 RequestTimeTooSkewed Error

Recently, we have started noticing “RequestTimeTooSkewed” erro while making requests to Amazon S3. The official FAQs suggests following:

Amazon S3 requires all machines making requests be within 15 minutes of an Amazon S3 webserver’s clock

Reading documentation and various blog posts made me believe, this is problem with the date/time settings on machine (which makes request). However, in our case, it turned out to be something else.

We use boto (Python library) for AWS stuff. Boto caches the connection objects (for S3, EC2, SQS, etc.), and tries to reuse those in later calls.

Imagine a simple use-case:

  1. Read data from S3
  2. Process data on EC2-instance/your-server
  3. Store processed-data on S3

In our case, #2 takes longer than 15 minutes (allowed limit by S3) sometimes, hence,  we were noticing ‘RequestTimeTooSkewed’ error.

Anyway, we fixed it by explicitly creating a different/fresh S3(http) connection before sending request to S3. This might sound inefficient, perhaps it is.

This is how we do now, a stripped down version of code from our class:

s3_connection = boto.connect_s3(aws_access_key_id="aws_access_key_id",aws_secret_access_key="aws_secret_access_key")
output_bucket = s3_connection.get_bucket("output_bucket")
key = output_bucket.new_key(key_name)
key.set_contents_from_filename(file_path, file_headers, True, None, 10, file_policy)

I am Python and boto noob; I am sure there would be a better way of doing this, so please share if you know one.