Remote Graphical Rendering

Remote Graphical Rendering

Joel Isaacson

Ascender Technologies Ltd.

Copyright 2012 Joel Isaacson

Ascender Technology Ltd Remote Rendering

Cellphone App


Application Level (1)

Ascender Technology Ltd

This is the normal way Linux/Android/Iphone runs apps.

The application itself is exported and run locally. Normally applications either come installed with

the system or are added later (e.g. Android Market).

Applications can run locally without using any network bandwidth.

Remote Rendering

Bitmap Level (4)


Common approach to exporting graphics (VNC, Chromoting).

Works well for any mix of graphics applications. Simple to implement. Heavy user of network bandwidth. Amenable to compression. Computationally intensive.

Remote Rendering

Toolkit Level (2)


It is technically very complex. Android, to date, has 15 different toolkit API variants.

Every application can extend the toolkit with custom widgets (subclasses of android.view.View).

Clearly impossible.

Remote Rendering

Android Rendering Level


This work exports graphics at the rendering level. In Android there are a number of rendering

interfaces that can be used:

1. Skia graphics

2. OpenGL ES 1.1 or OpenGL ES 2.0

3. Android.view.View We shall see that in effect toolkit and application

level graphics can be exported.

Remote Rendering

The Rendering Layer Expanded


Software Rendering Level (Skia)


The rendering level is the graphics layer that actually “colors” the pixels in the bitmap.

The Skia Graphics Engine is a compact open source graphics library written in C++.

It was originally developed by Skia Inc., which was subsequently acquired by Google, who released the software as open source.

Now known as Skia, it is currently used in Google Chrome, Chrome OS, Chromium OS, and Android.

Remote Rendering

Rendering Level (Skia)


Skia is a complete 2D graphic library for drawing Text, Geometries, and Images. Features include:

3x3 matrices w/ perspective Antialiasing, transparency, filters Shaders, xfermodes, maskfilters,

patheffects

Remote Rendering

Rendering Level (Skia)


Each Skia call has two components: the primitive being drawing (SkRect, SkPath, etc.) and color/style attributes (SkPaint).

For example:canvas.save();

canvas.rotate(45);

canvas.drawRect(rect, paint);

canvas.drawText(“abc”, 3, x, y, paint);

canvas.restore();

Remote Rendering

NinePatch Widget



GUI WidgetNinePatch Code

DrawbitmapRect(); // 1DrawbitmapRect(); // 2DrawbitmapRect(); // 3DrawbitmapRect(); // 4DrawRect()DrawbitmapRect(); // 5DrawbitmapRect(); // 6DrawbitmapRect(); // 7DrawbitmapRect(); // 8DrawbitmapRect(); // 9

Android Remote Rendering


Android Remote Rendering


We can simply modify the Skia library to export graphics by adding a small stub to Android to serialize and send a RPC-like rendering stream.

The local client can be on any system (Linux, MS Windows, Mac OS).

The prototype developed was on Linux-X11. Only 17 Skia routines had to be modified to

provide remote Android graphics.

Remote Rendering

Round Trip Latencies


The US east-west coast round trip is about 70 ms. Just one round trip delay per frame would reduce

the possible frame rate to 14 FPS. Potentially every Skia call returns a result. Some

results are critical for “Measure Passes”. In order to obtain measurements, we run a Skia

renderer in parallel on the remote machine. We can then send the rendering stream at full

network bandwidth, via a simplex link.Remote Rendering

Android Remote RenderingSimplex Network Connection


Compression


Android apps have a frame rate of 30-60 FPS. For every frame the complete rendering sequence

is sent. Compression is a good idea.

Remote Rendering

MPEG Compression


Video streams, photographically taken or photorealistically synthesized, can be compressed with MPEG standard codecs.

Even though no specific technique used in MPEG compression is applicable for the compression of our problem domain (rendering streams), some of the assumptions about how the data sets are generated are similar and the compression model design is similar.

Remote Rendering

MPEG Compression Techniques


The conversion of RGB images to YUV and sub-sampling is motivated by the color physiological opponency theory model of human vision.

Within each frame, the accuracy of spacial changes with shorter wavelengths are less important than the accuracy of longer wavelengths.

Inter-frame compression is based on finding motion vectors of the current frame based on previous and subsequent frames.

Remote Rendering

Rendering Stream Assumptions


The rendered material consists of a large number of sequential images (frames).

The target of the GUI images is the human visual system.

The subject matter while not being a product of our everyday visual world is modeled on the visual contexts of this world.

The GUI images are generated frame after frame by repeated invocation of GUI procedures.

Remote Rendering

Repeated Invocation of GUI Procedures


In practice, the number of unique sequences of rendering functions in execution paths taken within the code are bounded. This is because the rendering commands are generated by a fixed number of GUI functions and an application running a bounded amount of code.

The execution paths can be incrementally learned and entered into a procedure dictionary as the rendering commands are streamed.

Remote Rendering

Rendering Compression Techniques


Even if the sequences of rendering functions themselves are in the dictionary, the data arguments associated with these functions might be quite different from one another. Therefore, we keep a dictionary of the data arguments previously encountered for this particular sequence.

As a rendering procedural sequence with associated data is encountered, the data sequence dictionary for this procedural sequence is searched for the closest match to the current data sequence.

Remote Rendering

Android Contact List(Froyo)


Android Contact


Structured Procedural Compression


A careful examination of the rendering stream generated by the Android GUI reveals additional structural information that can be used to improve the data model.

The rendering stream has balanced save()-restore() pairs within each frame of the rendering stream.

Each save() is found at the beginning of a GUI function and a restore() is found at the end of each GUI function.

Remote Rendering



This information can be used to reverse engineer individual GUI and application procedures.

It will also reveal the call-graph of these procedures. Using this information, the rendering code

dictionary becomes a rendering procedure dictionary.

The call-graph data is best embedded in the per-procedure data dictionary.

DEMORemote Rendering



The server (encoder) constructs a “procedure dictionary” that is identical to the dictionary that is constructed by the client (decoder). Similar of LZW compression.

The procedures in this dictionary, surprisingly, are reversed engineered toolkit and application level routines.

We in effect have ended up exporting graphics at the toolkit-application level.

Remote Rendering

Structured Procedural CompressionStatistics


Our compression algorithm was tested on the rendering trace of a 60 frame sequence.

There were 13702 rendering commands for an average of 228 rendering commands per frame.

Of the 13702 rendering commands, there were 2691 functions (save/restore pairs).

Of these, only 47 were unique. This gives a compression rate of 1.75% (about 1:57).

Remote Rendering

Structured Procedural CompressionStatistics


Of the 13702 rendering commands only 354 had completely unique data parameter sets and 203 had data sets which are partially different.

Only these 557 data sets must be transmitted, thereby giving data compression of 4.06% (about 1:25).

If the partially different data sets are differentially transmitted, a data compression of 3.3% (about 1:30) is obtained.

Remote Rendering

Structured Procedural CompressionEntropy Encoding


For the last stage of compression, general techniques such as run length encoding (RLE) and entropy encoding (Huffman or Arithmetic) are used to produce a minimum bit count representation of the compressed material.

This additional phase should be expected to reduce the number of bits by a factor of 2-3

Remote Rendering

ICS GUI Rendering


Under ICS remote rendering can be done in any of three different interfaces:

1) Software rendering – Skia

2) Hardware rendering – OpenGL ES 2.0

3) Abstract rendering – Canvas class All three interfaces can use procedural

compression. The OpenGL rendering stream is much more

verbose (5x) than the equivalent Skia stream.

Remote Rendering

Cloud computing


Any Android app can be run on the server. It is not necessary that the remote server and the

local device have the same architecture, i.e. an ARM server can provide services for an Intel device.

For example: You might use Remote Rendering to provide Android apps to Chrome OS.

Remote Rendering

App Library / Subscription Model


Currently, apps are loaded into the local device - either installed at the time of purchase or added subsequently.

If efficient remote execution of apps can be supported, then instead of software purchases, a subscription model becomes possible.

A fixed monthly fee would entitle the subscriber to access a large library of applications.

Remote Rendering

App Demos


In a purchase/rental model, apps can be demoed remotely, prior to purchase.

If the user of the device finds the app to his/her liking, it can then be bought.

Remote Rendering

Remote Enterprise Applications


Applications may benefit from running within the enterprise's data centers, with the obvious benefits of scalability, security and maintainability.

Enterprise workers typically migrates through various computing environments (mobile-fixed-mobile) during a typical day.

Having the application running on a server will allow the running applications to migrate to various clients seamlessly.

Remote Rendering

Set-top Boxes


Many set-top boxes cannot practically run Android since the set-top box might not have sufficient resources or might not contain an ARM processor.

A local client application that only performs remote rendering takes limited resources and needs no long term persistent state.

Set-top boxes typically don't have complex local installations. Installing Android apps locally would present a support challenge to the operator.

Remote Rendering

Remote Graphical Rendering

Technology

Transcript of Remote Graphical Rendering