Graphics system

[joskuijpers]

We need to start making decisions about the graphics system.

Multiple ideas were offered. The one I remember was a Scene(Node) system, where every drawable object is a node with a parent and with children.
When a node rotates, so do the children. This is very easy to do using matrices. I like it.

Using matrices, any node can have a number of properties, such as rotation, shear, scale, relative position. This can all be in JS. Then there is a function createMatrix() that creates a 3x3 matrix for all these properties. A simple render would look like this:

Queue<Node> queue;

queue.enqueue(getRootNode());
while(!queue.isEmpty()) {
     Node *node;

     node = queue.dequeue();
     node.relativeMatrix = JavaScriptCall(node, "createMatrix");
     node.matrix = node.super.matrix * node.relativeMatrix; // or matr*relMat, I dunno.

     // use node.matrix to do transformation of node.vertices
     // draw the stuff

     queue.enqueue(node.children);
}

JS could keep track of changing properties and only recreate the matrix if something actually changes. Now, when setting transformation properties, everything is in JS, and to get all these properties, only 1 JS call is needed.
I have no idea how this will actually work out with OGL3 shaders.

@FlyingJester proposed something else, where shapes are not relative to each other. Maybe he could elaborate.

[FlyingJester]

My graphics API is as follows:

There are three basic types: Groups, Shapes, and Vertex's.

A group has a shader, an offset, and a rotation. A Shape has an offset, a rotation, and a texture. A vertex has a position and a color.

All vertices belong to a shape, and all shapes belong to a group (more on the 'all' part of that later).

To construct any primitive, you specify a shape by specifying vertices. The shape belongs to a group. Drawing a primitive (or anything at all) requires a valid shape, at least.

When a Shape is drawn, a polygon consisting of the vertices of the shape, textured by the shape's texture, and masked by the verticess' colors is drawn.
It is rendered/shaded by the shape's group's shader.
It is drawn at an offset, first by the shape's offset, then rotated by the shape's rotation. It is then offset by the group's offset and rotated by the group's rotation.

The only exception to all this is Shapes that have no group. They are child objects of a conceptual, immutable global group that has no offset and a rotation of 0 (ie, it is the window/screen).

If a shape has a group, it cannot be drawn on its own, the entire group must be drawn. If it has no group (aka, is part of the global group) it must be drawn on its own. The global group cannot be drawn all at once this way.

Another older description (which lacks offsets, rotations, and groups) can be found here: http://forums.spheredev.org/index.php/topic,1155.msg4623.html#msg4623

This provides basically the same functionality as what @joskuijpers proposed, but with different organization. Matrices would be expressed as groups, but in this case all groups are totally independant of each other. It also makes shaders specific to a group, which would make the Shader API very simple to implement on the backend.

You could think of this API as modern OpenGL 4 bound to JavaScript. At least conceptually :)

[FlyingJester]

@Radnen asked me (some time ago) about special graphics cases, specifically font.drawText() and its kin.

For text, I think it would be simplest to make text only renderable to surfaces. You would then have to dump the surface to an image and texture a shape with the text. The example on SphereDev demonstrates just that.
This is lets all the optimizations in the engine be realized in script. It also makes sense from a game makers perspective: how often do you draw text for a single frame only? Most other cases could be handled similarly. It keeps the hardware-based API very simple.

Thinking about plugin-based engines like TurboSphere, it also makes extending the graphics api with extra plugins a lot easier.

Other cases, like WindowStyles, are actually very well suited to this API. A WindowStyle is a group, each element is a shape (give or take).

[joskuijpers]

How well will this perform? Seeing we must minimize the calls between native and JS.

[FlyingJester]

It maps extremely well to OpenGL calls. In addition, after setting up a scene, the actual number of calls to draw it are at most the same as the Sphere 1.x API would require. In most cases it would be substantially less.

[joskuijpers]

@FlyingJester As you seem to have the system working, could you start writing a draft for Pegasus?

[FlyingJester]

Already available at https://github.com/sphere-group/pegasus/blob/master/api/ts/galileo.js
That is the exact API that TS exposes, minus two extensions:

• TS allows user-defined shaders
• TS allows Surfaces and Images to be blitted directly (really they just have an implicit shaped associated with them)

For the first, it requires the engine use GL at a very low level, which probably should not be necessary. The second is just to make the new API have a few shortcuts to make the long time Sphere users feel more at home--myself included!

[joskuijpers]

I suggest we create a sphere_ext_shaders for user defined shaders: I am willing to implement it too. For the surfaces: sphere_legacy_surface.

[FlyingJester]

We should be specific about these things. With shaders, I know that some hardware (ie, old OpenGL 2.0, some rather later Intel cards) can only do a single shading stage, but in TS I intend to allow at least a whole-scene shading in addition to per-group shading. Surfaces still exist, but they take on a much more appropriate role, being used as a software graphics buffer with some graphical capabilities rather than just more advanced, slightly slower to blit Images as in Sphere 1.x.
We should be specific about the surface extension. Here's what I recommend:
sphere_ext_global_shaders
sphere_ext_group_shaders
At the moment I define the surface blitting extension in TS as:
sphere_legacy_surface_direct_blit

Note how some OpenGL extensions are labelled:
https://developer.apple.com/library/mac/documentation/GraphicsImaging/Conceptual/OpenGLExtensionsGuide/Reference/reference.html

Usually the only time you name an extension after an object, the extension indicates the object's existence. When the extension is some relaxation of the standard limitations (for instance, GL_ARB_texture_non_power_of_two) it's usually the name of the object or procedure in question, and then what it is. For extensions, it's usually a very brief description and then what system it extends.

[fatcerberus]

For Surfaces, I think it might be more useful to specify them as an abstraction over FBOs (render to texture) rather than as a software buffer. The latter can be implemented using a plain JS ArrayBuffer (note that minisphere can already create Images from ArrayBuffers as of v3.0), so there's no real call for an additional dumb buffer type in the core API.

[FlyingJester]

The main difference is the access performance characteristics. FBOs are of course much slower to read from than software buffers. Very often, I used Surfaces when I wanted to intersperse reads and writes, for actions such as anti-aliasing or blurring, or adding graphical effects. This would have been much, much slower if the Surfaces were purely FBOs.

Of course, the API isn't much geared for one or the other more, so it's not too important where they are actually stored from a semantics perspective.

[fatcerberus]

Regarding Galileo: In my tests, I've found the current setup of the Group and Shape objects to be limiting. I'd propose that Shapes and Groups have no explicit location and rotation and instead have all transformations to be handled by a matrix. To make this easier on the user, the Group could have designated methods for the various transforms, for example

group.identity();  // or maybe group.reset if that's clearer
group.translate(tx, ty);
group.rotate(angle);
group.scale(sx, sy);
// etc.

If you need direct access to the transformation matrix, you would access the .matrix property. In this way the Group encapsulates both the shaders used to draw the primitive(s) as well as the full set of transformations in a very generic way.

The one downside to this, of course, is that matrix multiplication is not reversible (in the general case at least). So you'd lose the ability to read the X and Y coordinates, or the angle of rotation, in isolation. But since a Group will likely be a member of a larger class which does know its location (like an NPC class, say), this shouldn't be a real issue in practice.

[joskuijpers]

@fatcerberus I am not against this idea. I do think however that we then would need a Matrix class with transformation functions. And that it is purely functional.
So: x = new Matrix(4,4); y = x.multiply(x); That means y is a copy of x. x did not change. Why you want this? In order to make group.matrix actually useful. You could opt for a clone() function but that is just horrible semantics. Then you, as a JS programmer, needs to think about pointers and such.

[fatcerberus]

No no, Matrix is not a value type: it is mutable. So if you do this:

group.matrix.translate(1, 1);

The group will move over and down by 1 pixel. I think this is fine: JS programmers expect objects to have reference semantics. I think I'd find the purely functional approach to be more strange, honestly.

[joskuijpers]

Alright.

If you want, we could chat (in IRC?) about pegasus.

[fatcerberus]

I wonder if it would be worthwhile to have an optional module that provided GL bindings to JS code directly? OpenGL is very powerful but it has a stateful design that is difficult to map to any kind of object-oriented interface without hurting performance. Most drivers seem to be optimized for you not swapping in and out textures/shaders/buffers/etc constantly but that's exactly what ends up happening with Galileo.

[joskuijpers]

That will probably be too slow as well, because of the switches between JS and native.

[FlyingJester]

Then you run into the issues of what a certain card does and does not support being put upon script to determine.

Galileo shouldn't be switching shaders that often, unless you interleave groups with different shaders. In fact, in almost all cases, the mapping between OpenGL and a Galileo call is very close, and so if you have the knownledge to optimize OpenGL calls you know more than enough to optimize Galileo calls.

[fatcerberus]

That's true. If Galileo is the only graphics API available you can optimize it not to change states as often. In the current minisphere implementation I end up switching in and out shaders between the Galileo shader and the Allegro built-in one constantly, for every Group, because they have different requirements (notably texture coordinates are interpreted differently), but if the legacy functions aren't needed natively that wouldn't be necessary.

[FlyingJester]

This was a primary reason behind dropping the old API in TurboSphere. It seems pretty clear that to get real hardware acceleration, and to really make good use of it in a simple and intuitive way, it cannot coexist with the old primitive functions. The old drawing API is just too deeply rooted in the immediate-style of functionality.

[fatcerberus]

For Galileo I wonder if it would make sense to rename Group to Model, since that's really what a group conceptually represents, a 2D/3D model made up of primitives (Shapes). minisphere currently uses ShapeGroup for the purposes of avoiding name collision, but having a two-word class name seems awkward when almost everything else is one word (Image, Surface, Mixer, etc.)