ubuntu-developer-manual team mailing list archive

[Rick Spencer <rick.spencer@xxxxxxxxxxxxx>]

Please find attached my first draft of the Multimedia chapter.

Cheers, Rick

= Introduction =
This chapter will introduce the basics of adding multi-media to your applications. We'll start by creating a simple image viewer application, and then change that application to support overlays. We'll finish by adding the ability to play movies to the application.

To prepare, create an application using $quickly create, and delete the boiler plate widgets as per usual. 

<screenshot of application, ready to start>

= Displaying an Image =
For the first iteration, we will display and image from the user's Pictures directory, assuming the user has added a Picture there called "portrait.jpg". The simplest way to present an image is to use a gtk.Image widget. A gtk.Image widget is a container for a gtk.gdk.pixbuf. the gtk.gdk.pixbuf containts the actual image data. Add a gtk.Image to an application using Glade. Since the gtk.Image has no data to start, the "broken image" icon will display by default.

<Image with broken image>

Using glib.get_user_special_dir() to find a path to Pictures directory ensures that the application works well even after it's been translated or if the Pictures directory is moved for some reason. 

First, import the glib library:
import glib

Then add the following code to the finish_initializing function to create a path 
To access standard directories, use glib.get_user_special_dir() like this:

        picture_dir = glib.get_user_special_dir(glib.USER_DIRECTORY_PICTURES)
        path = os.path.join(picture_dir, "portrait.png").

Then set the gtk.Image to display that picture:
        self.ui.image1.set_from_file(path)

<picture of running application>
But as you can see, the image isn't displayed very well. So we should change the size of the picture before displaying it. But since the image is just a widget for image data in a gtk.gdk.PixBuf, we don't change the size of the gtk.Image, we have to load the picture as a gtk.gdk.PixBuf, and then resize it, and then add it into the image. So the following code will load the picture data, scale it, and set the gtk.Image to use that data:
        #load the pixbuf
        picture_dir = glib.get_user_special_dir(glib.USER_DIRECTORY_PICTURES)
        path = os.path.join(picture_dir, "portrait.jpg")
        pixbuf = gtk.gdk.pixbuf_new_from_file(path)

        #calculate new sizes for the pixbuf
        new_width = 600
        new_height = 600
        if pixbuf.get_height() > pixbuf.get_width():
            new_width = int((float(new_height)/pixbuf.get_height()) * pixbuf.get_width())
        elif pixbuf.get_width() > pixbuf.get_height():
            new_height = int((float(new_width)/pixbuf.get_width()) * pixbuf.get_height())
    
        #scale the pixbuf and use the scaled image
        scaled_buf = pixbuf.scale_simple(new_width,new_height,gtk.gdk.INTERP_BILINEAR)
        self.ui.image1.set_from_pixbuf(scaled_buf)

Now when you run the app, you can see the image is scaled properly.

<Screenshot of running application>

= GooCanvas =
Rather than just displaying the picture, let's say we want to display a little Ubuntu badge on the image, and a title. For composing with images and graphics, we use a goocanvas.Canvas object with some related classes. GooCanvas is a widget that allows mashing up all kinds of graphics, like pictures, shapes, text, paths, etc... For just displaying a picture, it's a bit more complex than using a gtk.Image, but it's a lot easier for mashing up graphics.

== Add a goocanvas.Canvas ==
Delete the gtk.Image from Glade, and then back in the code, start out by importing goocanvas:
import goocanvas

Also, go ahead and delete the pixbuf scaling code, and the code that adds the pixbuf to the gtk.Image. So delete this code:
        #scale the pixbuf and use the scaled image
        scaled_buf = pixbuf.scale_simple(new_width,new_height,gtk.gdk.INTERP_BILINEAR)
        self.ui.image1.set_from_pixbuf(scaled_buf)

The goocanvas library isn't integrated into Glade, so you need to create the widget and add it on the fly.
        #create and add the goocanvas
        canvas = goocanvas.Canvas()
        canvas.set_size_request(500,400)
        canvas.show()
        self.ui.vbox1.pack_start(canvas)

Now when you run the app, you can see the blank goocanvas.Canvas:
<insert picture of app with blank goocanvas>

== Add an Image to the goocanvas.Canvas ==
We already wrote the code to load the image from the file, as well as to calculate height and width to make it fit. So it's a matter of creating a goocanvas.Image object, and associating that object with the goocanvas.Canvas. Note that when you create the goocanvas.Image, you tell it that it is associated with the goocanvas.Canvas, rather than adding it to the canvas.


        #load the pixbuf
        picture_dir = glib.get_user_special_dir(glib.USER_DIRECTORY_PICTURES)
        path = os.path.join(picture_dir, "portrait.jpg")
        pixbuf = gtk.gdk.pixbuf_new_from_file(path)

        #calculate new sizes for the pixbuf
        new_width = 500
        new_height = 400
        old_width = pixbuf.get_width()
        old_height = pixbuf.get_height()
        if old_height > old_width:
            new_width = int((float(new_height)/old_height) * old_width)
        elif old_width > old_height:
            new_height = int((float(new_width)/old_width) * old_height)

        #add the image
        root_item = canvas.get_root_item()
        image = goocanvas.Image(parent=root_item, pixbuf=pixbuf)

When you run this code, you can see that the image is added:
<insert picture of huge image>

But of course, the size is not set. For a goocanvas.Items that you add to a goocanvas, you don't the height and the width, you scale the image. So a small calculation is required to turn the new_width and new_height variables into the proper scales, before calling the scale function:
        w_scale = float(new_width)/old_width
        h_scale = float(new_height)/old_height
        image.scale(w_scale, h_scale)

Now, the image is scaled to the proper size.
<insert picture of nicely scaled image>

== Add a Badge to the Image ==
Next we'll place a Ubuntu badge on the image, using the file "background.png". In a quickly project, media files like images and sounds should always go into the data/media directory so that when users install your programs, the files will go to the correct place. There is a helper function called get_media_file built inot quickly projects to get a URI for any media file in the media directory. You should always use this function to get a path to media files, as this function will work even when your program is installed and the files are put into different places on the user's computer. get_media_file returns a URI, but a pixbuf expects a normal path. It's easy to fix this stripping out the beginning of the URI. Since it was created for you, can could also change the way get_media_player works, or create a new function, but this works too:

        logo_file = helpers.get_media_file("background.png")
        logo_file = logo_file.replace("file:///","")
        logo_pb = gtk.gdk.pixbuf_new_from_file(logo_file)

Then add the logo to goocanvas.Canvas in the same way you added the main image, except set the x and y for the position of the logo when creating it.
        logo = goocanvas.Image(parent=root_item, pixbuf=logo_pb,x=2200, y=1550)
        logo.scale(.2,.2)

<picture of image with logo>
You may notice that x and y for the logo are on a different scale than for the main image, the canvas is 500 by 400 pixels, 
but the logo is visible even though it is set at 2200 X 1550. This is because goocanvas.Image is a goocanvas.Item,and goocanvas.Items maintain their own coordinate systems even when you do things like scale and rotate them. There are two functions to help you convert between different scales. To convert a point on a canvas to a point in an item's scale use the the canvas's  convert_to_item_space funcion. So to find out what the point 100 x 100 on the canvas is to in an logo's space:
item_x, item_y = canvas.convert_to_item_space(logo, 100,100)

You can tranlsate back using canvas.convet_from_item_space. So to find the position 50 x 50 for the logo on the canvas:
canvas_x, canvas_y = canvas.convert_from_item_space(logo, 50, 50).

== Add Text to an Image ==
To add an title to the picture, use the goocanvas.Text class. It works in a similar manner to the goocanvas.Image class:
        title = goocanvas.Text(parent=root_item,text="Ilsabe", x=120, y=10)
        title.set_property("font","Ubuntu")
        title.scale(2,2)

<picture of image with title showing>

The goocanvas library affords you the flexibility to be very creative and easily do lots of fun things in your applications. There are different kinds of Items and many of interesting visual things you can do with them. There are items like shapes and paths. You can change things like their scale, rotation, and opacity. You can even animate them!

The whole block of code for adding the image, logo and text looks like this:
        #create and add the goocanvas
        canvas = goocanvas.Canvas()
        canvas.set_size_request(500,400)
        canvas.show()
        self.ui.vbox1.pack_start(canvas)
 
        #load the pixbuf
        picture_dir = glib.get_user_special_dir(glib.USER_DIRECTORY_PICTURES)
        path = os.path.join(picture_dir, "portrait.jpg")
        pixbuf = gtk.gdk.pixbuf_new_from_file(path)

        #calculate new sizes for the pixbuf
        new_width = 500
        new_height = 400
        old_width = pixbuf.get_width()
        old_height = pixbuf.get_height()
        if old_height > old_width:
            new_width = int((float(new_height)/old_height) * old_width)
        elif old_width > old_height:
            new_height = int((float(new_width)/old_width) * old_height)

        #add the image
        root_item = canvas.get_root_item()
        image = goocanvas.Image(parent=root_item, pixbuf=pixbuf)

        #scale the image
        w_scale = float(new_width)/old_width
        h_scale = float(new_height)/old_height
        image.scale(w_scale, h_scale)
    
        #read in and add the logo file
        logo_file = helpers.get_media_file("background.png")
        logo_file = logo_file.replace("file:///","")
        logo_pb = gtk.gdk.pixbuf_new_from_file(logo_file)
        logo = goocanvas.Image(parent=root_item, pixbuf=logo_pb, y=1550,x=2200)
        logo.scale(.2,.2)

        #add a title
        title = goocanvas.Text(parent=root_item,text="Ilsabe", x=120, y=10)
        title.set_property("font","Ubuntu")
        title.scale(2,2)

= Using the Web Cam =
You can easily add Web Cam functionality to your application using quickly.widgets.WebCamBox. 

== Add a WebCamBox ==
Start by importing the WebCamBox:
from quickly.widgets.web_cam_box import WebCamBox

Then create and add a WebCamBox in the finish_initializing function:
        self.cam = WebCamBox()
        self.cam.show()
        self.cam.set_size_request(400,300)
        self.ui.vbox1.pack_start(self.cam)

If you run the application now, the web cam doesn't appear.
<picture of blank window>

This is because the camera has not been started. It's important to give the application a chance to realize all it's widgets before you try to run the WebCamBox because it might not be ready to start painting, and the camera won't start. You can add a button to start it playing, but you can also use gobject.timeout_add to tell it to start after a certain period of time, for instance, to start it playing after half a second (500 milliseconds), start by importing the gobject library:
import gobject

Then create a function to start the camera. It's important for the fucntion to return False, otherwise, timeout_add will call it over and over again.
    def play_web_cam(self):
        self.cam.play()
        return False

And then add this to your code after creating the WebCamBox in finish_initializing to tell it to start:
        gobject.timeout_add(500, self.play_web_can)
<picture of webcam working>

This calls the WebCamBox's play() function after 500 milliseconds. As you may have guessed, you can stop the camera using the WebCamBox's stop() function.

== Taking a Picture ==
The WebCamBox has a built in function called "take_picture". This takes a picture from the camera, names it with the current time stamp, and drop's it in the Pictures directory. 

So maybe a fun app would start up, take a picture, and then quit.

First, create a function to take the picture and then stop the camera:
    def take_picture_from_cam(self):
        self.cam.take_picture()
        self.cam.stop()
        self.destroy()
        return False

Then add a line to call that function:
        gobject.timeout_add(3000, self.take_picture_from_cam)

Here's all the code to make the web cam start, and take a picture, and then stop:
        #create a web cambox
        self.cam = WebCamBox()
        self.cam.show()
        self.cam.set_size_request(400,300)
        self.ui.vbox1.pack_start(self.cam)

        #run functions after a period of delay
        gobject.timeout_add(500, self.play_web_cam)
        gobject.timeout_add(3000, self.take_picture_from_cam)

    def play_web_cam(self):
        """play_web_cab - starts the web cam.

        """

        self.cam.play()
        return False

    def take_picture_from_cam(self):
        """take_picture_from_cam - taks a picture, and quits the program.
   
        """

        self.cam.take_picture()
        self.cam.stop()
        self.destroy()
        return False

The take_picture function returns a path to the picture it took, in case a program wants to do some processing with that picture. This tweak will simply print the path of the picture to the terminal:
        path_to_pic = self.cam.take_picture()
        print path_to_pic

== Using GStreamer ==
A MediaPlayerBox is a wrapper around a GStreamer library's camerabin object. GStreamer gives you access to significant power and flexibility in editing and modifying the output of the webcam. A MediaPlayerBox as a camerabin property, so simply use cam.camerabin to get a reference to the gstreamer camerabin.

= Playing Media =
You can easily add the ability to play video or sound files to your application using quickly.widgets.MediaPlayerBox. A MediaPlayerBox works just was well for sound or video.

== Add a MediaPlayerBox ==
Adding a MediaPlayerBox is a lot like adding a WebCamBox. Start by importing MediaPlayerBox:
from quickly.widgets.media_player_box import MediaPlayerBox

Then in the finish initializing function, create and pack it into the window:
        self.player = MediaPlayerBox(True)
        self.player.show()
        self.player.set_size_request(600,450)
        self.ui.vbox1.pack_start(self.player)

When creating a MediaPlayerBox, by default, it has no controls. However, you can pass in True when creating MediaPlayerBox to tell it to show controls by default. If you run the app, you can see the controls, but of course, they don't work because there is no file for the application to play.
<screenshot of application with controls visible>

Rather than a reference to file, MediaPlayerBox uses a URI to know what to play. So, in order to use he movie "minecraft.avi" in the Video director, create a path, and then turn it into a well formed URI before setting the MediaPlayerBox's property:

        #create the URI
        video_dir = glib.get_user_special_dir(glib.USER_DIRECTORY_VIDEOS)
        path = os.path.join(video_dir, "minecraft.avi")
        uri = "file://" + path

        #Set the URI for the player
        self.player.uri = uri

Now when you run the application you can use the play button to play and pause the video, and you can use the scroller to control the position.
<screenshot of application with mediaplayer showing>

== Controling the MediaPlayerBox ==
A MediaPlayerBox is designed to be easy to control from your program. So you can play, pause, and stop media playing using functions:
        self.player.play()
        self.player.pause()
        self.player.stop()

You can also get the current position in the media file in secconds:
    current_second = self.player.position

You can change the position, too. To jump to the first minute of the video:
        self.player.position = 60

You can toggle whether the controls show:
        self.player.controls_visible = True

== Knowing When a File is Done ==
It's often useful to know when a media file is done playing, for example to load another file, or to loop the file. To set up looping, or autoreplaying of a file, start by creating a replay funcion.

    def replay(self, widget, data=None):
        self.player.stop()
        self.player.play()
        self.player.position = 0

Then in finish_initializing, after you create the MediaPlayerBox, connect the replay function to the end of file signal:
        self.player.connect("end-of-file",self.replay)

Now the video will replay when it reaches the end. Here's all the code for looping the minecraft.avi video:
        #create a MediaPlayerBox
        self.player = MediaPlayerBox(True)
        self.player.show()
        self.player.set_size_request(600,450)
        self.ui.vbox1.pack_start(self.player)

        #create the URI
        video_dir = glib.get_user_special_dir(glib.USER_DIRECTORY_VIDEOS)
        path = os.path.join(video_dir, "minecraft.avi")
        uri = "file://" + path

        #Set the URI for the player
        self.player.uri = uri
        self.player.connect("end-of-file",self.replay)

    def replay(self, widget, data=None):
        self.player.stop()
        self.player.play()
        self.player.position = 0

== Using GStreamer ==
Just like the WebCamBox wraps a camerabin, MediaPlayerBox wraps a Gstreamer playbin. If you want to access it, just use the player.playbin property.