OOPSLA Talk on Preon
-
Upload
wilfred-springer -
Category
Technology
-
view
2.101 -
download
4
description
Transcript of OOPSLA Talk on Preon
![Page 1: OOPSLA Talk on Preon](https://reader033.fdocuments.us/reader033/viewer/2022050816/54b4d81a4a795994558b45a0/html5/thumbnails/1.jpg)
Bit Syntax for JavaPreon
![Page 2: OOPSLA Talk on Preon](https://reader033.fdocuments.us/reader033/viewer/2022050816/54b4d81a4a795994558b45a0/html5/thumbnails/2.jpg)
Preon
A declaratve data binding framework for binary encoded data
![Page 3: OOPSLA Talk on Preon](https://reader033.fdocuments.us/reader033/viewer/2022050816/54b4d81a4a795994558b45a0/html5/thumbnails/3.jpg)
Binary Encoded Data?
Any fle for which 'file -bi {filename}' does not return a mimetype startng with 'text/'
Non-binary:text/plain (*.txt)text/html (*.html)text/xml (*.xml)
Binary:applicaton/pdf (*.pdf)applicaton/octet-stream (*.mp4)image/png (*.png)
![Page 4: OOPSLA Talk on Preon](https://reader033.fdocuments.us/reader033/viewer/2022050816/54b4d81a4a795994558b45a0/html5/thumbnails/4.jpg)
Not only byte stream
Always octects (8 bit values)
8 bits8 bits
![Page 5: OOPSLA Talk on Preon](https://reader033.fdocuments.us/reader033/viewer/2022050816/54b4d81a4a795994558b45a0/html5/thumbnails/5.jpg)
But also bit stream
Always octects (8 bit values)
5 bits 5 bits
![Page 6: OOPSLA Talk on Preon](https://reader033.fdocuments.us/reader033/viewer/2022050816/54b4d81a4a795994558b45a0/html5/thumbnails/6.jpg)
Decoding TomTom Map Files
Approx. 300 pages of C++ source code, just for decoding only...
![Page 7: OOPSLA Talk on Preon](https://reader033.fdocuments.us/reader033/viewer/2022050816/54b4d81a4a795994558b45a0/html5/thumbnails/7.jpg)
Compressed Data
2 kg
1021 pages
![Page 8: OOPSLA Talk on Preon](https://reader033.fdocuments.us/reader033/viewer/2022050816/54b4d81a4a795994558b45a0/html5/thumbnails/8.jpg)
Challenges
● Decoding from bit stream not for the faint-hearted
● Encoding to bit stream not for the faint-hearted
● Hard to maintain
● Hard to extend
● Java doesn't help (Bug 4504839)
● Decoder and encoder easily go out of sync
● Documentaton and sofware easily go out of sync
![Page 9: OOPSLA Talk on Preon](https://reader033.fdocuments.us/reader033/viewer/2022050816/54b4d81a4a795994558b45a0/html5/thumbnails/9.jpg)
What if....
..this could all be solved easily?
![Page 10: OOPSLA Talk on Preon](https://reader033.fdocuments.us/reader033/viewer/2022050816/54b4d81a4a795994558b45a0/html5/thumbnails/10.jpg)
Preon Ambitions
● Declaratvely map data structure to encoded representaton
● Get the decoder/encoder/documentaton, free of charge
![Page 11: OOPSLA Talk on Preon](https://reader033.fdocuments.us/reader033/viewer/2022050816/54b4d81a4a795994558b45a0/html5/thumbnails/11.jpg)
5-Second User Guide
01 File file = new File(...);02 Codec<BitMap> codec = Codecs.create(BitMap.class);03 BitMap bitmap = Codecs.decode(codec, file);
Decoding a BitMap:
![Page 12: OOPSLA Talk on Preon](https://reader033.fdocuments.us/reader033/viewer/2022050816/54b4d81a4a795994558b45a0/html5/thumbnails/12.jpg)
What just happened?
01 File file = new File(...);02 Codec<BitMap> codec = Codecs.create(BitMap.class);03 BitMap bitmap = Codecs.decode(codec, file);
Create a Codec for instances of BitMap
... and use it to decode a BitMap.
![Page 13: OOPSLA Talk on Preon](https://reader033.fdocuments.us/reader033/viewer/2022050816/54b4d81a4a795994558b45a0/html5/thumbnails/13.jpg)
Find the specification
The data structure IS the specifcaton:
class BitMap { @Bound int width; @Bound int height; @Bound int nrColors; @BoundList(size=”nrColors”) Color[]; @BoundList(size=”width*height”) byte[] pixels;}
class Color { @Bound int red; @Bound int green; @Bound int blue;}
![Page 14: OOPSLA Talk on Preon](https://reader033.fdocuments.us/reader033/viewer/2022050816/54b4d81a4a795994558b45a0/html5/thumbnails/14.jpg)
Demo
![Page 15: OOPSLA Talk on Preon](https://reader033.fdocuments.us/reader033/viewer/2022050816/54b4d81a4a795994558b45a0/html5/thumbnails/15.jpg)
But what is actually happening?
Codec is nothing but a facade to a chain of Codecs.
Each Codec only understands its own task.
Codecs will delegate to other Codecs
![Page 16: OOPSLA Talk on Preon](https://reader033.fdocuments.us/reader033/viewer/2022050816/54b4d81a4a795994558b45a0/html5/thumbnails/16.jpg)
While Decoding
![Page 17: OOPSLA Talk on Preon](https://reader033.fdocuments.us/reader033/viewer/2022050816/54b4d81a4a795994558b45a0/html5/thumbnails/17.jpg)
Codecs
Encapsulate everything there is to know about the mapping between in-memory representaton and encoded representaton
![Page 18: OOPSLA Talk on Preon](https://reader033.fdocuments.us/reader033/viewer/2022050816/54b4d81a4a795994558b45a0/html5/thumbnails/18.jpg)
Demo Generated Documentation
![Page 19: OOPSLA Talk on Preon](https://reader033.fdocuments.us/reader033/viewer/2022050816/54b4d81a4a795994558b45a0/html5/thumbnails/19.jpg)
So, now....
Forget everything I just told you
![Page 20: OOPSLA Talk on Preon](https://reader033.fdocuments.us/reader033/viewer/2022050816/54b4d81a4a795994558b45a0/html5/thumbnails/20.jpg)
Preon just works
Annotatons
● @Bound● @BoundList● @BoundString● @BoundNumber● @BoundObject● @BoundExplicitly● @If ● @LazyLoading● @LengthPrefx● @TypePrefx● @ByteAlign● @Slice● @Import
Types
int, byte, short, long, Integer, Byte, Short, Long, boolean, Boolean, String, List<T>, T[], type-safe enums, Object
Expressions
atributes: .{name}items: [{number}], or [{expr}]arithmetc: +, -, /, *, ^combinatorial: &&, ||relatonal: >=, <=, ==, <, >literals: 'foobar', 0x0f, 0b01101
![Page 21: OOPSLA Talk on Preon](https://reader033.fdocuments.us/reader033/viewer/2022050816/54b4d81a4a795994558b45a0/html5/thumbnails/21.jpg)
Convention over Configuration
// Default settings for int Codec: // 32-bits, little endian@Bound int foo;
// Read an int, but construct the int // from 5 bits only@BoundNumber(size=”5”) int foo;
![Page 22: OOPSLA Talk on Preon](https://reader033.fdocuments.us/reader033/viewer/2022050816/54b4d81a4a795994558b45a0/html5/thumbnails/22.jpg)
Expressions (1)
/** * An icon of maximal 15 * 15 pixels. Each color * has a color in the range 0-255. */public class Icon {@BoundNumber(size=”4”) int height;@BoundNumber(size=”4”) int width;@BoundList(size=”height * width”) byte[] pixels;}
By acceptng Strings instead of booleans and integers, Preon allows you to pass in expressions.
![Page 23: OOPSLA Talk on Preon](https://reader033.fdocuments.us/reader033/viewer/2022050816/54b4d81a4a795994558b45a0/html5/thumbnails/23.jpg)
Expressions (2)
@BoundString(size=”{expr}”, ...)
@BoundList(size=”{expr}”, offset=”{expr}”, ...)
@BoundNumber(size=”{expr}”, ...)
@Slice(“{expr}”)
...
![Page 24: OOPSLA Talk on Preon](https://reader033.fdocuments.us/reader033/viewer/2022050816/54b4d81a4a795994558b45a0/html5/thumbnails/24.jpg)
Boolean expressions
@Boundprivate int mapVersion;
@If(“mapVersion >= 700”)@Bound private MapFlags flags;
// But also:// mapVersion + 1 >= 700// mapVersion > 3 && mapVersion < 300// etc.
![Page 25: OOPSLA Talk on Preon](https://reader033.fdocuments.us/reader033/viewer/2022050816/54b4d81a4a795994558b45a0/html5/thumbnails/25.jpg)
References
Backward references only
Reference the “outer” object
addresses[1].streetouter.driversLicenses[1].statedriveresLicenses[nrDriverLicenses - 1].state
![Page 26: OOPSLA Talk on Preon](https://reader033.fdocuments.us/reader033/viewer/2022050816/54b4d81a4a795994558b45a0/html5/thumbnails/26.jpg)
Variable Introductions
@Bound int[] offsets;@BoundList(offset=”offsets[index]”,...) List<Node> nodes;
● Preon will inject List implementaton
● List implementaton will load Nodes lazily, on demand
● Calling nodes.get(3) will cause the List implementaton to
– Calculate the node's positon: ofsets[index=3] = 120
– Call Codec<Node> to start decoding from that point
Introducton
![Page 27: OOPSLA Talk on Preon](https://reader033.fdocuments.us/reader033/viewer/2022050816/54b4d81a4a795994558b45a0/html5/thumbnails/27.jpg)
Inheritance
Java Classes Preon Perspectve
![Page 28: OOPSLA Talk on Preon](https://reader033.fdocuments.us/reader033/viewer/2022050816/54b4d81a4a795994558b45a0/html5/thumbnails/28.jpg)
Codecs class is your friend
static <T> T decode(Codec<T> codec, byte[] buffer)static <T> T decode(Codec<T> codec, ByteBuffer buffer)static <T> T decode(Codec<T> codec, File file)
static <T> Codec<T> create(Class<T> type) static <T> Codec<T> create(Class<T> type, CodecFactory...
factories)static <T> Codec<T> create(Class<T> type, CodecDecorator...
decorators)
static <T> void document(Codec<T> codec, ArticleDocument document)
static <T> void document(Codec<T> codec, DocumentType type, OutputStream out)
static <T> void document(Codec<T> codec, DocumentType type, File file)
![Page 29: OOPSLA Talk on Preon](https://reader033.fdocuments.us/reader033/viewer/2022050816/54b4d81a4a795994558b45a0/html5/thumbnails/29.jpg)
If not Preon, then what else?
Declaratve approach is not new:– BSDL (Bitstream Syntax Descripton Language)– Flavor (htp://favor.sourceforge.net/)– XFlavor– BFlavor (htp://multmedialab.elis.ugent.be/bfavor/)
None of them would worked in our case:– Overly complicated– No free/solid implementaton– Assumptons are surreal:
● “everything will just ft in memory”● “there will only be a single thread”● “our current feature set is all you ever need”
![Page 30: OOPSLA Talk on Preon](https://reader033.fdocuments.us/reader033/viewer/2022050816/54b4d81a4a795994558b45a0/html5/thumbnails/30.jpg)
The Preon Answer
“everything will just ft in memory”
Preon is loading data lazily by default, and when it does it relies on memory mapped data (Default BitBufer implementaton wraps around ByteBufer, and hence also MappedByteBufer.)
“there will only be a single thread”
In Preon, every thread can have its own reference to the current positon in the BitBufer.
“our current feature set is all you ever need”
Preon has incredibly open: implement CodecFactory or CodecDecorator to create your own codecs based on type informaton or annotatons.
![Page 31: OOPSLA Talk on Preon](https://reader033.fdocuments.us/reader033/viewer/2022050816/54b4d81a4a795994558b45a0/html5/thumbnails/31.jpg)
CodecFactory
interface CodecFactory { <T> Codec<T> create(AnnotatedElement metadata, Class<T> type, ResolverContext context);}
Annotatons on the object expectng data to be decoded by the Codec.
The type of object to be decoded.
The object for constructng references.
returns null if it does not have a way to construct a Codec
![Page 32: OOPSLA Talk on Preon](https://reader033.fdocuments.us/reader033/viewer/2022050816/54b4d81a4a795994558b45a0/html5/thumbnails/32.jpg)
Chain of CodecFactories
![Page 33: OOPSLA Talk on Preon](https://reader033.fdocuments.us/reader033/viewer/2022050816/54b4d81a4a795994558b45a0/html5/thumbnails/33.jpg)
CodecFactory Usage
● CodecFactories can be passed to Codecs.create(...)
● ... which will cause the Codecs class to consider these factories when constructng the various Codecs.
● Internally, a big chain of commands
![Page 34: OOPSLA Talk on Preon](https://reader033.fdocuments.us/reader033/viewer/2022050816/54b4d81a4a795994558b45a0/html5/thumbnails/34.jpg)
Current Status
Preon 1.0 released
(htp://preon.fotsam.nl/)
Big Theme for next Release:
Encoding
![Page 35: OOPSLA Talk on Preon](https://reader033.fdocuments.us/reader033/viewer/2022050816/54b4d81a4a795994558b45a0/html5/thumbnails/35.jpg)
Other Future Work
● Improved debugging
● Annotaton driven support for other compression techniques
● More hyperlinking in the documentaton
● Beter algebraic simplifcaton of expressions
● “In-place” editng?
![Page 36: OOPSLA Talk on Preon](https://reader033.fdocuments.us/reader033/viewer/2022050816/54b4d81a4a795994558b45a0/html5/thumbnails/36.jpg)
Conclusions
● It can be done!!!
● Breaking down Codecs into smaller codecs results into some interestng propertes
● The ideas in Preon could work in other Java data binding frameworks as well
● Encoding and decoding bitstream compressed data is stll hard, but Preon truly hides that complexity
● Preon is interestng enough to deserve more support
![Page 38: OOPSLA Talk on Preon](https://reader033.fdocuments.us/reader033/viewer/2022050816/54b4d81a4a795994558b45a0/html5/thumbnails/38.jpg)
Preon Layers
An expression language capable of rendering itself to human readable text. (limbo.sourceforge.net)
A fuent interface for generatng documents.(pecia.fotsam.nl)
BitBufer abstractons
Data binding
![Page 39: OOPSLA Talk on Preon](https://reader033.fdocuments.us/reader033/viewer/2022050816/54b4d81a4a795994558b45a0/html5/thumbnails/39.jpg)
Preon License
Apache 2.0
Apache 2.0
GPL + Classpath Excepton
GPL + Classpath Excepton