Snowmix Glshapes Guide

Introduction.

GL Shapes or glshapes enables use of OpenGL in Snowmix for video mixing. With OpenGL Snowmix can create 2D and 3D effects possibly hardware accelerated. Hardware acceleration require the hardware to support OpenGL on the system Snowmix is running. If hardware acceleration is not supported, Snowmix will fall back to Off Screen CPU based rendering for OpenGL support. Glshapes is alpha code in Snowmix version 0.5.0.

OpenGL 3D example

Overview

• Including Glshapes Support.
• GL Context.
• GL Shapes.
• GL Textures.
• GL Vectors.
• GL Lights and Materials.
• Vertex Buffer Objects.
• Vertex Array Objects.
• Placed GL Shapes.
• Copying Back From the GPU.
• OpenGL Links and Ressources.

OpenGL has not been tested on the ARMv7 platform. Most implementations with ARMv7 offers OpenGL ES as opposed to regular OpenGL. Quite likely only few changes to Snowmix will be necessary to get Snowmix to work with OpenGL ES, but this is currently untested.

Including Glshapes Support.

Because Glshapes is alpha-code in Snowmix version 0.5.0, you need to compile or recompile Snowmix to include support for Glshapes. To do this you will need to execute the following commands:

$ cd TO_WHERE_THE_SNOWMIX_CODE_TREE_RESIDES $ ./bootstrap $ ./configure --enable-snowmixosmesa --enable-snowmixx11 --enable-snowmixglu --prefix=/usr/local $ make clean $ make $ sudo make install

You should now have installed a Snowmix binary that supports Glshapes. To check this, you can start Snowmix and execute the order gl info and see some information on the support of Glshapes. However note that information on OpenGL version etc. is not available until a glshape has been overlayed or to be more precise, not until a OpenGL context has been created.

GL Context.

The GL Context is an OpenGL context within Snowmix enabling OpenGL graphics. Snowmix supports two kind of OpenGL contexts namingly the OSMesa and the GLX/X-Windows. The OSMesa context is CPU based. The GLX context is GPU assisted and requires a local graphic adapter supporting OpenGL. Further more the local machine has to support and run X-Windows as well as allowing Snowmix to open a window within its X-Window session.

Using a GPU can offer great performance, not only because a GPU can have a very high performance executing otherwise complicated heavy load CPU tasks, but also because it can do so in parallel with the CPU perhaps doing other tasks. However it comes at a cost of not only complexicity and the cost of learning a new skill being OpenGL programming, but it also usually require transferring large amount of data from CPU memory to GPU and back again. While the transfer to the GPU is DMA and queue based, transferring from GPU to CPU memory is currently CPU intensive.

You select which context you want to use by including the command glshape context in your ini file or be executing it through a control connection. The command should be used before the first glshape is overlayed. The syntax of the command is:

glshape context [(osmesa|glx|auto)]

The actual context is created when the first placed glshape is overlayed/executed. When the context is created, the glshape id 1 is executed, if it exist. After the context is created, then for every frame period the first time a placed glshape is overlayed/executed, the glshape id 2 is executed, if it exists.

In Snowmix version 0.5.0, the OSMesa context is using the Snowmix main mixer frame as its drawing context while the GLX context is using buffers in the GPU. For the OSMesa context this means that whatever you have mixed/drawn in the main mixer frame before overlaying a placed glshape, the context will draw ontop of that. The GLX context is different. Not only will you for that context have to copy data from the main mixer frame to the GPU, which can be done using textures, but you will also have to copy the result back to either the main mixer frame or to a loaded image for later usage.

Note to Mac OS X users

For Snowmix version 0.5.0, the OSMesa Context may not work. This may just be a linking issue, but the root cause has not been found yet.

GL Shapes.

Snowmix allows defintion of glshapes being a list of primarily OpenGL graphic primitives/commands similar to creating list of Cairo Graphics primitives/commands for regular Snowmix Shapes.

A GL shape is defined by its unique glshape ID. Each glshape needs a glshape place holder defined with the command glshape add. This is shown in the example below:

glshape add 1 My first glshape

Here the glshape place holder 1 is defined and given the name 'My first glshape'. After the glshape has been created, it can now be added commands to its list. The commands are primarily OpenGL primtives with only a few exceptions. The commands all have the form:

glshape COMMAND_NAME <shape id> ......

Such commands will add the command to end of the glshape command list identified by the shape id. It is important to understand that commands added to a glshape are parsed and added to a glshape command list, but they are not executed at that moment. For the commands in a glshapes command list to be executed, a glshape has to be placed and overlayed. Below is an example of creating a glshape command list with commands:

glshape add 1 My first glshape glshape loadidentity 1 glshape matrixmode 1 projection glshape enable 1 blend

The glshape command list can be inspected using the command

glshape list <glshape id>

as shown below:

gl list 1 STAT: shape 1 ops 3 name My first glshape STAT: - 1 loadidentity STAT: - 2 matrixmode projection STAT: - 3 enable mode blend STAT:

The command glshape moveentry can be used to move commands within a glshape command list as well as used for deleting a specific command in a glshape command list.

The command glshape modify can be used to change numbers and setting of some commands in a glshape command list.

The command glshape entry can be used to set the entry active or inactive or set active for a limited number of frames/references. When an entry is inactive, it will be skipped when the shape is executed.

If the glshape 1 is later executed, that is if the glshape is placed and overlayed, the following OpenGL C/C++ commands are excuted in whatever context is active at that moment:

• glLoadIdentity();
• glMatrixMode(GL_PROJECTION);
• glEnable(GL_BLEND);

A glshape definition can later be used as the basis for a placed glshape like a regular shape is used for a placed shape in Snowmix.

Please also see the section GL Shape Commands on the page Reserved Commands.

Special Shapes

There a few special glshape IDs. These are 0, 1 and 2. Glshape ID 0 is reserved for internal on-the-fly creation use. The glshape ID 1 is executed, if it exists, when the OpenGL Context is created. See the GL Context section for specifics. The glshape ID 2 is executed, if it exists, once for every frame imidiately before the first glshape is overlayed/executed and only then. The idea for glshape ID 1 is to place commands in that glshape you wish to have executed upon initialization of the context. Such commands could include setting up projection, coordinate system etc. The idea for glshape ID 2 is to place commands in that shape you want to have executed for each frame period such as clearing the drawing buffer before starting drawing with OpenGL primitives.

GL Textures.

Textures for Glshapes in Snowmix are images the shader can use as a surface on 2D and 3D objects. Snowmix supports 2D and Cube textures. The source of textures in Snowmix are images, video feed frames and the main mixer frame. Textures are used to move image data from CPU memory into the graphical context or GPU.

Defining a texture

To use a texture in Snowmix, a place holder within Snowmix has to be created. This is done using the command glshape texture add. Each texture place holder is identified by its unique texture Id, which is an integer number starting from 0. In the following example, 3 textures are created. First the place holders for the textures are created:

glshape texture add 1 Texture for Mixer Frame glshape texture add 2 Texture for Video Feed #1 glshape texture add 3 Texture for Loaded Image #5

The syntax for the command is as follows:

glshape texture add [<texture id> [<texture name>]]

The command texture add can not be added to a glshape command list although technically it is possible to workaround this limitation.

A texture place holder is deleted using the command glshape texture add with only the texture id argument.

Setting the source for a texture

Before using a texture, the source of the texture must be defined. This is done using the command glshape texture source. In the example below, the source is set for the 3 texture place holder we just defined above:

glshape texture source 1 frame 0 glshape texture source 2 feed 1 glshape texture source 3 image 5

In this example we defined texture 1 to be sourced by the mixer frame 0 (only one exist). We also set texture 2 and 3 to be sourced by feed 1 and image 5 respectively. Feed 1 and image 5 must exist at this point. The source of a texture can be removed setting the source to none as shown in this example:

glshape texture source 2 none

The syntax for the texture source command is as follows:

glshape texture source [<texture id> (feed <feed id>|frame <frame id>|image <image id>)|none [<id>]]

There is no simple OpenGL equivalent to the glshape texture source command. The command glshape texture source can not be added to a glshape command list although it is possible to workaround this limitation.

Binding a texture

The actual moving of image data from CPU memory to the context or GPU is not taking place until a texture is bound. This is done using the command glshape texture bind. The command needs to part of a glshape command list and that glshape has to be placed and overlayed. Below is shown the full syntax for the glshape texture bind command:

glshape texture bind <glshape id> <texture id> [<min> <mag>] [2d | cube]

When a glshape is executed, meaning it is referenced within a placed glshape and that placed glshape is overlayed, and then it reaches the texture bind command, it will allocate an internal texture in the context, for the GLX context in the GPU. Such a ressource is limited by the amount of RAM available for the context/GPU. After an internal texture has been generated, the image data is loaded to the texture in the context possibly the GPU. The texture bind allocation is equivalent to the the following OpenGl commands:

• glGenTexture();
• glBindTexture();
• glTexImage2D(); or gluBuild2DMipmaps(GL_TEXTURE_2D...);
• glTexParameter(GL_TEXTURE_2D, GL_TEXTURE_MIN_FILTER, min);
• glTexParameter(GL_TEXTURE_2D, GL_TEXTURE_MAG_FILTER, mag);

In the following example glshape 1 is set to create a texture, source it, bind it and use it on a four sided polygon, a quad. This glshape can then later be placed and overlayed. When a placed glshape is overlayed, its command list is executed line by line. See the following example with line numbers included for understanding:

1 : glshape add 1 This is a list of shape commands 2 : glshape enable 1 texture_2d # Enable 2D textures. 3 : glshape texture bind 1 3 linear nearest # Bind texture 3 as current texture 4 : glshape begin 1 quads # Begin a 4 point polygon 5 : #glshape normal 1 0 0 1 # Setting the normal for the quad, here commented out 6 : glshape texcoord 1 0 0 # Using current texture lower left 7 : glshape vertex 1 -0.5 -0.5 0 # Setting Lower left corner 8 : glshape texcoord 1 1 0 # Using current texture lower right 9 : glshape vertex 1 0.5 -0.5 0 # Setting Lower right corner 10 : glshape texcoord 1 1 1 # Using current texture upper right 11 : glshape vertex 1 0.5 0.5 0 # Setting Upper right corner 12 : glshape texcoord 1 0 1 # Using current texture upper left 13 : glshape vertex 1 -0.5 0.5 0 # Setting Upper left corner 14 : glshape end 1 # Ends the definition of the quad polygon 15 : glshape disable 1 texture_2d # Disabling the 2D Texture feature

The first line defines a glshape place holder with the Id 1 and the name 'This is a list of shape commands'.

The second line enables the OpenGL feature TEXTURE_2D using glEnable\(GL_TEXTURE_2D);.

The third line defines texture 3 to be bound as the current texture. When the shape list is executed, this is the line that will initiate image data to e send from the CPU memory to the context or GPU. See above for the relation between the texture bind command and OpenGL commands.

In the 4th line a definition of a quad polygons is defined. In this case we only define one, but it could contain definition of more than one. A valid begin definition must later end with an end definition. The command is equivalent to the OpenGL command glBegin(GL_QUADS).

The 5th line is defining the normal of the quad defined. The normal is a vector usually perpendicular to plane of the quad. In this case, the normal definition is commented out. The command is equivalent to the OpenGL command glNormal(x,y,z).

The next four vertex commands defines the four 3D (x,y,z) coordinates of the quad. Note that the points are defined counter clockwise. The commands are equivalent to glVertex(x,y,..).

Before each vertex command is a texcoord command defining how the texture is intended to be merged onto the quad. Each texcoord commands takes a set of 2d coordinates s and r, where s=0 means the left side of the image and r=0 means the bottom of the image. These values are fraction of the image and must be between 0 and 1. The OpenGL equivalent is glTexCoord(s,r,..).

In the 14th line we have the corresponding end command to the previous begin command. The OpenGL equivalent is glEnd().

Finally the command list ends by disabling the feature TEXTURE_2D. The OpenGL equivalent is glDisable\(GL_TEXTURE_2D);.

The above defined shape can be listed using the command gl list 1:

gl list 1 STAT: shape 1 ops 13 name This is a list of shape commands STAT: - 1 enable mode texture_2d STAT: - 2 texture bind id 3 min linear mag nearest type 2d : Texture 3 Texture for Loaded Image #5, image 5 STAT: - 3 begin quads STAT: - 4 texcoord2 0.0000,0.0000 STAT: - 5 vertex3 -0.5000,-0.5000,0.0000 STAT: - 6 texcoord2 1.0000,0.0000 STAT: - 7 vertex3 0.5000,-0.5000,0.0000 STAT: - 8 texcoord2 1.0000,1.0000 STAT: - 9 vertex3 0.5000,0.5000,0.0000 STAT: - 10 texcoord2 0.0000,1.0000 STAT: - 11 vertex3 -0.5000,0.5000,0.0000 STAT: - 12 end STAT: - 13 disable mode texture_2d STAT:

The above example can be rewritten into two shapes adding flexibility:

1 : glshape add This a Quad shape with texcoords 2 : glshape begin 1 quads # Begin a 4 point polygon 3 : #glshape normal 1 0 0 1 # Setting the normal for the quad 4 : glshape texcoord 1 0 0 # Using current texture lower left 5 : glshape vertex 1 -0.5 -0.5 0 # Setting Lower left corner 6 : glshape texcoord 1 1 0 # Using current texture lower right 7 : glshape vertex 1 0.5 -0.5 0 # Setting Lower right corner 8 : glshape texcoord 1 1 1 # Using current texture upper right 9 : glshape vertex 1 0.5 0.5 0 # Setting Upper right corner 10 : glshape texcoord 1 0 1 # Using current texture upper left 11 : glshape vertex 1 -0.5 0.5 0 # Setting Upper left corner 12 : glshape end 1 # Ends the definition of the quad polygon 13 : glshape add 2 This is a glshape including another shape 14 : glshape enable 2 texture_2d # Enable 2D textures. 15 : glshape texture bind 2 3 linear nearest # Bind texture 3 as current texture 16 : glshape inshape 2 1 # include glshape 1 17 : glshape disable 2 texture_2d # Disabling the 2D Texture feature

The benefit is that it is quite easy to change the glshape included in line 16 to another glshape with another outline or otherwise other settings.

Please also see the section GL Shape Commands on the page Reserved Commands.

GL Vectors.

In Snowmix Vector constants can be created (and changed) using the commands glshape vector add and glshape vector value. The vectors can have 2, 3 and 4 dimensions. Internally the vectors are saved as either integers, floats or doubles depending on the settings for compiling Snowmix. The vectors are needed when defining light and materials. The syntax for the two vector commands are:

glshape vector add [<vector id> [<vector name>]] glshape vector value [<vector id> <x> <y> [<z> <w>]]]

Both the add and the value commands when used without argument, will list the created vectors name and values respectively. If the add command is used with the vector id argument, but no name is given, the vector with that id will be deleted. The maximum number of vectors Snowmix can have, is currently fixed and set in the file src/opengl_video.h.

The GL vectors have no direct equivalent in OpenGL, but several OpenGL commands are taking an array of values as arguments, which is what we are doing in Snowmix too.

Please also see the section GL Shape Commands on the page Reserved Commands.

GL Lights and Materials.

Yet to be written.

Defining light constants

Yet to be written.

Light commands

Yet to be written.

Please also see the section GL Shape Commands on the page Reserved Commands.

Vertex Buffer Objects.

The OpenGL commands glBegin and glEnd are deprecated in OpenGL 3.0 though still supported. In OpenGL 3.1 the commands has been removed. For now the commands will stay in Snowmix. Instead of glBegin/glEnd a Vertex Buffer Object or VBO for short must be created and used. The benefit of VBO over the older glBegin/glEnd method is that the data points for Vertex only has to be sent to the GPU once and whenever they change if ever. An additional benefit of using a VBO over the glBegin/glEnd commands is that in a VBO the data points can be reused thus lowering the number of data points needed to be send from CPU to GPU. Reusing datapoints is possible when a VBO defines more than one polygon and at least two polygons share one or more datapoints.

Previous a Quad Vertex with texture was defined using the glBegin/glEnd equivalent. Here the same quad is being defined as a VBO:

glshape vbo add 5 Quad with texture glshape vbo config 5 static quads t2 v3 glshape vbo data 5 0.0 0.0 -0.5 -0.5 0.0 1.0 0.0 0.5 -0.5 0.0 1.0 1.0 0.5 0.5 0.0 0.0 1.0 -0.5 0.5 0.0 glshape vbo indices 5 0 1 2 3

The first line adds/creates a VBO place holder with the vbo_id 5 and the name "Quad with texture".

The second line sets up the VBO 5 as a static VBO containing one or more quad polygons with the elements t2 and v3 meaning a 2-dimensional texture (glTexCoord) on a 3-dimensional vertex (glVertex3). This means that each dataset in the VBO consist of 5 data points. The first two datapoints are the texture datapoints and the next 3 are the vertex datapoints.

The third line defines for VBO 5 four datapoint sets.

The fifth line defines defines how the VBO should use the datapoint sets by defining an indexing. What is says is that when drawing/executing the VBO, the context should use the datasets in the order 0, 1, 2, 3. In this case, there is no benefit using indexing as there is only defined one quad. However if we add a second plane into the z-plane that share the right side of the first quad, we can add extra index points:

glshape vbo data 5 0.0 0.0 0.5 -0.5 1.0 0.0 1.0 0.5 0.5 1.0 glshape vbo indices 5 1 4 5 2

Now when the VBO is used, two four-sided polygons sharing two points are drawn. First the datasets 0,1,2,3 are used for the first quad and then the datasets 1,4,5,2 are used for the second polygon. In this case we saved 2 data sets at the expense of extra index information. The benefit will increase with the number of adjacent polygons sharing datapoints.

In a glshape a defined VBO can be referenced as shown in the following reusing the example in GL texture section (see the GL Texture section):

13 : glshape add 2 This is a glshape including another shape, well now we include a vbo instead 14 : glshape enable 2 texture_2d # Enable 2D textures. 15 : glshape texture bind 2 3 linear nearest # Bind texture 3 as current texture 16 : glshape vbo 2 5 # include VBO 5 17 : glshape disable 2 texture_2d # Disabling the 2D Texture feature

When glshape 2 is overlayed, it will reference VBO 5. If this is the first time the VBO is referenced, the VBO's datapoints will be sent to the GPU. In future references when overlaying a glshape containing VBO 5, fewer OpenGL calls are needed and less data has to be sent from CPU to GPU.

Vertex Array Objects.

VBOs in Snowmix are, if supported by the OpenGL library on the platform, automatically wrapped in a Vertex Array Object or VAO so when overlaying a glshape containing a VBO, all Snowmix has to do is binding the VAO containing the VBO. However the first time the VBO is referenced in a placed glshape, the VBO is first defined for the VAO sending the VBO data to the GPU. It all happens automatically and is done with only a couple of OpenGL calls minimizing CPU load significantly.

Adding a functionality to combine several VBOs in one VAO is under consideration.

VAOs seems to be supported on Ubuntu 14.04 LTS on a Thinkpad T61, but the result may for you depend on your graphic driver. The command gl info reveals for Ubuntu 14.04 on a Thinkpad T61 the following about the OpenGL driver for the GLX Context and the OSMesa Context respectively:

... STAT: OpenGL vendor : Intel Open Source Technology Center STAT: OpenGL renderer : Mesa DRI Intel(R) 965GM STAT: OpenGL Version : 2.1 Mesa 10.1.3 STAT: OpenGL Shading : 1.20 STAT: OpenGL vendor : Brian Paul STAT: OpenGL renderer : Mesa OffScreen STAT: OpenGL Version : 2.1 Mesa 10.1.3 STAT: OpenGL Shading : 1.20 ...

The Mesa 3D has in late July 2015 landed the updates for Mesa 10.7 that adds full support for OpenGL 4.0 and most of 4.1 and 4.2. Mesa 10.7 is planned for release as Mesa 11.0 in September 2015. However the state of the hardware drivers are lacking behind. You can read more here. Furthermore, these updates will take some time to filter down into official Linux distro releases.

On a MacbookPro Late 2011 with Yosemite 10.10.4, VAOs seems not to be supported and Snowmix falls back to using VBOs without wrapping them in a VAO.

Please also see the section GL Shape Commands on the page Reserved Commands.

Placed GL Shapes.

Created glshapes can be placed and later overlayed. The command to place a glshape is the glshape place command. The command has the following syntax:

glshape place [<place id> [ <shape id> <x> <y> <z> [<scale x> <scale y> <scale z> [<rotation> <rx> <ry> <rz> [<red> <green> <blue> [<alpha>]]]]]

If used without any arguments, it list all placed glshapes. If used with just the <place id> argument, it will delete the place holder of the placed glshape, if it exists. If used with both a <place id> argument and <shape id> argument and a 3D coordinate as argument, it will create a place holder for a placed glshape with the id <place id>.

Note that the coordinate system will completely depend on how you setup and use glshapes. By default, positive Y is up, which is the opposite of the Cairo Graphic Context in Snowmix.

In addition to the arguments already explained, you can also specify the scale, rotation, color and alpha of the placed glshape. The rotation is different from rotation for regular placed shapes in Snowmix, because it in addition to its angle, also needs a direction or vector in the form of a 3D coordinate specifying the direction of the angle.

Snowmix has a number of commands for manipulating settings for a placed glshape. You can list these commands by issuing the command glshape place help on a Snowmix control connection.

Placed glshapes can be overlayed/executed using the command glshape overlay. However this command, as overlay commands for texts, images, feeds and shapes, needs to be executed as part of the command macro set with the command overlay finish. The syntax of the overlay command is:

glshape [ place ] overlay (<id> | <id>..<id> | <id>..end | all) [(<id> | <id>..<id> | <id>..end | all)] ....

Please also see the section Placed GL Shape Commands on the page Reserved Commands.

Copying back from the GPU.

If you are using the GLX context and if you have overlayed/executed placed glshapes, you will want to copy back the result to either a loaded image or to the main mixer frame. The OSMesa context for Snowmix 0.5.0 in contrast to the GLX context, does not need that because it is drawing in the main mixer frame, which may change later. Copying back for the GLX context is done using the command glshape copyback. This command will have to part of the command macro that is set with the Snowmix command overlay finish. The syntax of the command is:

glshape copyback [frame 0 | image <image_id> <x> <y> <width> <height>]

If you for the command use no argument or the frame 0 as arguments, then the drawing buffer of the GPU is copied back to the Snowmix main mixer frame overwriting it. That is probably what you want if the context is the GLX context. If your context is the OSMesa context, you overwrite the Snowmix main mixer frame with it self which is unnecessary and can have unpredictable results.

If you are using the image parameter including <image id> and more, you can write the GPU drawing buffer of the GPU back to a loaded image and use it for further mixing. If the loaded image did not exist, it will be created. If it did exist, it would be deleted and a new is created.

Copying back from the GPU is rather CPU intensive and should only be used when absolutely necessary, but of course, if you use glshapes in the GLX context, copying back will be required. Also remember that bandwidth between GPU and the computers main memory is a limited ressource. So if there are frame periods, where you don't generate any data in the GPU, copying back should be avoided.

OpenGL Links and Ressources.

Using glshapes for Snowmix requires understanding of how to use OpenGL. The following are links to OpenGL ressources that can be usefull to obtain the required knowledge.

• OpenGL Tutorials with video and examples.
• OpenGL Tutorials on opengl-tutorial.org..
• OpenGL.org.
• OpenGL at Khronos.
• Learning Modern 3D Graphics Programming
• OpenGL® on Silicon Graphics® Systems