The Case for Context-Aware Compression

Xuan Bao Trevor Narayan Ardalan Amiri Sani Wolfgang RichterDuke University Duke University Rice University Carnegie Mellon University

Romit Roy Choudhury Lin Zhong Mahadev SatyanarayananDuke University Rice University Carnegie Mellon University

ABSTRACTThe proliferation of pictures and videos in the Internet is im-posing heavy demands on mobile data networks. This de-mand is expected to grow rapidly and a one-fit-all solution isunforeseeable. While researchers are approaching the prob-lem from different directions, we identify a human-centricopportunity to reduce content size. Our intuition is that hu-mans exhibit unequal interest towards different parts of a con-tent, and parts that are less important may be traded off forprice/performance benefits. For instance, a picture with theStatue of Liberty against a blue sky may be partitioned intotwo categories – the semantically important statue, and theless important blue sky. When the need to minimize band-width/energy is acute, only the picture of the statue may bedownloaded, along with a meta tag “background: blue sky”.Once downloaded, an arbitrary “blue sky” may be suitably in-serted behind the statue, reconstructing an approximation ofthe original picture. As long as the essence of the picture is re-tained from the human’s perspective, such an approximationmay be acceptable. This paper attempts to explore the scopeand usefulness of this idea, and develop a broader researchtheme that we call context-aware compression.

1. INTRODUCTIONMobile broadband traffic continues to increase at anoverwhelming pace. Predictions for 2014 suggest a 39fold increase in demand, far exceeding the wireless ca-pacity promised by foreseeable technologies, such as4G/WiMax/LTE [1]. This dramatic increase is not onlyattributed to the surge in device density, but also to theeruption of high-resolution pictures and videos in the In-ternet. In fact, a study reports that by 2012, 3G networkswill become saturated if 40% of its subscribers consumevideo just for 8 minutes a day [2]. Network operatorsare aware of the impending crisis, and are beginning toadopt precautions. For instance, ATT has already rolledout tiered pricing schemes that require users to operatebelow a pre-specified download quota. The expectationis that users will forcibly curb their browsing habits andcollectively reduce the strain on the wireless spectrum.While pricing is indeed one solution to the problem, it

may not be the desirable one.

Several researchers have taken up the challenge to copewith mobile data demands, and are exploring ways tooffload cellular data networks. Ongoing approaches aremostly at the PHY/Link layer, including opportunisticmigration of 3G traffic to WiFi [3], the use of femtocells [4], smarter antennas [5], etc. We break awayfrom these schemes and explore a complimentary ap-proach that attempts to reduce the volume of content,without significantly reducing the user’s satisfaction.

Our observation is that humans are the dominant con-sumers of online content, and they exhibit an unequaldegree of interest for different parts of the content. Asan example, in a video of a stand-up comedy show, thecomedian’s actions may need to be preserved as is, how-ever, the backdrop may be amenable to modification.Similarly, in a picture of a child sitting in a garden (Fig.1), the garden may be altered without compromising thesatisfaction of the person who views the picture. If oneis able to isolate the subject of the content (the child)from its background (the garden), it may be possible toonly download the subject along with a brief descrip-tion of the background. Once downloaded, the receivercan select a similar background (from its local databaseof pictures) and carefully insert it behind the subject.The outcome is a variant of the original picture but isexpected to preserve its semantic value/context. Someinformation will obviously be lost, and may reduce theuser’s satisfaction. However, the cost savings in band-width and battery power may adequately compensatefor the dissatisfaction. If future network services comewith stricter price plans, we believe that context-awarecompression may offer a useful knob to cope with theprice-performance tradeoff. A person with little left inher download quota may opt for compressed news sites,where the picture of the President is received as is; onlythe large crowd he is addressing gets locally synthesized.

1.1 Natural QuestionsA number of questions arise as one begins to considerthe notion of context-aware compression. We touch

1

Figure 1: Core idea in context-aware compression

upon some of them here, and revisit them in subsequentsections through measurements and hypotheses testing.

(1) If the background is not important, why not performa (very) lossy compression on the background? This willpreclude the need to carefully replace the background witha different one. We believe that significantly loweringthe fidelity of the background may degrade the user’sviewing experience. The user may strive to discern whatthe background originally was – a blurred garden maynot be obvious after heavy lossy compression. Moreover,striking disparity in the resolution of the foregroundand background may make the picture look unnatural.With our proposal, the synthesized background remainsin high fidelity – the user immediately knows that thechild is in the garden. The background can also bewatermarked so the user can precisely learn about thesynthesized parts, and request the original if desired.

(2) Context-aware compression may not be acceptable al-ways. What kind of use-cases lend themselves to suchmodifications? Clearly, our ideas become less relevantwhen the viewer has adequate bandwidth, energy, orcan tolerate latency in consuming the content (e.g.,downloading over WiFi at home). Context-aware com-pression is also inappropriate for applications such asopportunistic search of crowd-sourced images. The en-tire premise there is that one user’s “background” mayin fact contain precious information of latent interest toanother user. Moreover, special-occasion pictures withinclose social circles (e.g., wedding pictures) will also notbe suitable for compression. However, not-so-specialcontent shared with broader audiences may offer oppor-tunity for context-aware compression. Applications mayinclude downloading news/blogs/articles over mobilephones, talk shows on mobile TV, music videos, etc. Thecontent providers may provide both compressed and un-compressed versions; users may choose one based ontheir position on the price-performance tradeoff.

This paper explores the theme of context-aware contentby defining the various hypothesis that needs to hold,and verifying them through small scale experiments. Weshow that humans indeed exhibit preferential treatmenttowards different parts of a content, and such prefer-

ences are quite correlated across individuals. Encour-aged by these findings, we develop a simple heuristicthat automatically identifies objects in a picture, andpreserves them during transmission. The background tothese objects are eliminated and replaced at the receiverusing templates created in PhotoShop. Although ourprototype is crude at this stage, we believe there is evi-dence that context-aware compression can be a relevantsoftware primitive for the future. The relevance will notonly increase with greater sophistication in image pro-cessing, but also with a stricter need to reduce contentfootprint for overloaded wireless networks and batteries.

The next section formulates and tests the hypothesesthat constitute the basis for context-aware compression.Thereafter, we present some preliminary heuristics thatdemonstrate the promise of this space. We close the pa-per with discussions on the longer term research agenda,followed by related work and a brief conclusion.

2. HYPOTHESES AND VERIFICATIONWe state 3 main hypotheses (in this paper, we focus onimages alone and treat videos as a time-sequence of im-ages). These 3 hypotheses are not meant to be exhaus-tive; they are the critical ones necessary to erect thetheme of context-aware compression.

1. Some parts of images are semantically less valu-able than others, and a user is willing to compro-mise the fidelity of these parts in exchange for per-formance gains. Let us call these less-valuable partsbackgrounds, and the complimentary portions (i.e.,the semantically valuable areas), foregrounds. Syn-thesizing the backgrounds (e.g., inserting a tem-plate garden behind the child) will not diminish thehuman’s satisfaction excessively.

2. Human users exhibit high overlap in their de-scription of image foregrounds and backgrounds.Therefore, a “good” partition of foreground andbackground will satisfy the majority of users.

3. Removing the background reduces the contentsize. If the background of a picture is naturallyamenable to heavy compression (e.g., a clear bluesky), then the gains from context-aware compres-sion will be negligible. We hypothesize that manypictures have a sizable background, and therefore,eliminating them during transmission is gainful.

In an attempt to verify these hypotheses, an experimentwas designed and conducted with real users. The exper-iment methods and findings are described next.

2.1 Experiment MethodologyWe implemented a simple image cropping tool in Java.The tool downloads random pictures from Flickr (or any

2

other source), draws a N × N grid on the picture, anddisplays it to the user. Users can select the background ofthe picture by selecting multiple grid boxes – the back-ground can be composed of multiple non-overlappingportions of the picture. We invested effort to make thebackground selection simple so that operational biasesare minimized. Once a user has selected the background,she performs a crop operation, which leaves only theforeground on the screen. The user has the option torevert to the original and make changes, if she feels thatthe context of the picture is not adequately captured.Figure 2 shows a screenshot from our software tool.

Figure 2: Software tool screenshot

We recruited 6 student volunteers, explained our ideasto them, and asked them to partition the foreground andbackgrounds in a way that would preserve the context ofthe picture. We asked them to imagine that the pictureswill be viewed over a mobile phone/iPad, and that theviewer is under a 200MB data plan (recently launchedby ATT). The images were selected randomly from Flickrand covered different genres, including natural scenes,street views, people, paintings, etc. Each participantcropped 50 pictures.

After the experiments, we interviewed the participantsto understand the reasonings behind their choices. Wespecifically intended to learn whether they were ableto satisfactorily crop out the essence of the pictures, asopposed to selecting only the visually appealing parts.Most people said that the “context in the pictures heavilyoverlapped with the visually appealing parts”. However,they emphasized that in several cases visually unattrac-tive parts were also selected because they were integralto the context. One user, for instance, said that “the ferryboat in front of the statue of liberty was not visually at-tractive ... yet, I included it to capture the tourism aspect”.Based on these interviews, we gained reasonable confi-dence that the foregrounds reflect the contexts.

2.2 Measurement ResultsWe verify the hypotheses based on the results of the ex-periments described above.

(H1) Some parts of images are semantically less valu-able than others. Figure 3 shows the CDF of the ra-tio between the foreground-area and the entire imagearea. Evidently, for more than 80% of the Flickr pic-tures, less than 70% of the image areas were croppedout as foreground. For around 50% of these pictures, theforeground covers less than 50% of the area of the en-tire image. This demonstrates that, on average, the lessvaluable parts of the picture – the background – makesup a reasonably large area of the picture. Synthesizingthem carefully can offer gains.

Figure 3: Ratio between the area of the foregroundand the area of the entire image

(H2) Human users exhibit high overlap in their de-scription of image foregrounds and backgrounds. Toverify whether different users attach importance to sim-ilar parts of the image, we computed the overlap in fore-ground for each pair of users. We define overlap as:

Foregroundi ∩ ForegroundjForegroundi ∪ Foregroundj

Foregroundi denotes the area of the foreground se-lected by user i. This equation compares the foregroundarea selected by both users with the foreground areaselected by at least one user. Figure 4 shows that in 50%of the cases, the overlap is around 75%. This supportsthe observation that humans’ perception of importanceare similar to each other.

Figure 4: Foreground “overlap” for all pairs of users.

(H3) Removing the background reduces the contentsize. Figure 5 shows the CDF of uncompressed andcompressed background file sizes, as cropped out by

3

the users in our experiments. The uncompressed back-ground is a JPEG file, while the compressed backgroundwas produced by subjecting the same JPEG file to thestandard Linux-based bzip operation. The two curvesexhibit a small gap between them implying that back-grounds are not significantly amenable to compression.

Figure 5: Compressed and uncompressed back-ground file sizes

We translate the above findings into expected perfor-mance gains. We assume that the background of eachpicture can be substituted by a high level meta-tag (e.g.,garden, cloudy-sky, etc.), and a similar background canbe inserted at the receiver. Thus, the performance gainG is the ratio of the background size to the entire sizeof the picture. If G = 0.4, it implies that context-awarecompression reduces the size of the picture by 40%. Wealso compute Gz, which is the gain if all pictures werezipped at the 3G tower, and unzipped at the mobile de-vice. We define Gz as bzip(background)

bzip(background)+bzip(foreground) .Figure 6 plots the distribution of G and Gz for all back-grounds cropped out by the human users. On average,picture sizes can be reduced by 50% if one is willing tounzip the content at the receiver (hence, pay an energycost). Otherwise, the savings are around 40%.

Figure 6: Potential gain for compressed and uncom-pressed images

3. FRAMEWORK DESIGNWe build a preliminary framework to demonstrate thefeasibility of context-aware compression. We target boththe image and the video domains.

3.1 Context-Aware Image CompressionWe have made an early attempt at context-aware im-age compression. An application that requires humanassistance on a per-image basis is not quite feasible. To-wards an automatic means of extracting the foreground,we employ a simple heuristic. We assume that the con-text of an image is typically captured through objects inthe image, and that these objects are often located nearthe center of the image. Thus, we first employ objectrecognition methods in image processing and accord-ingly identify the foreground.

We borrow object identification techniques from [6]. Au-thors in [6] use a combination of multiscale saliency,color contrast, edge density, and super-pixel straddling,to identify square shaped windows in an image. Ourheuristic selects large sized windows, located near thecenter, as the foreground. Figure 7 shows the opera-tion on a few randomly chosen Flickr images, used inour experiments. The union of the red-boxed areas is as-sumed to be the foreground. Our results show that thecomputer selected foregrounds reasonably overlap withthe human-selected foregrounds (Figure 8). More than50% of the cases, the overlap between two selected fore-grounds is more than 55%. We concede that our heuris-tic is not accurate – the selected foreground may notinclude all important objects and the boundary of theforeground may not align precisely with objects’ edges.

Figure 7: Images with selected foreground3.2 Video TransmissionFor many video content, the video-recording is per-formed in indoor environments, using cameras frommultiple vantage points. Examples include talk shows,news broadcast, interviews, stand-up comedies, etc. Inthese programs, the background environment is typi-

4

Figure 8: Overlap between foregrounds selected bycomputer heuristics and human users

cally (well-decorated) studio walls, stage backdrops, orperhaps a sitting audience behind the speaker. Whena video plays, the backgrounds across different video-frames are likely to be different views of similar environ-ment. Therefore, it may be possible to use templates forthe backgrounds for different room settings and differ-ent camera angles. For instance, we can have templatesfor the broadcasting room for different camera anglesand ranges. These templates can be pre-loaded at theusers’ mobile phones. Later, the video server can transferthe foreground with pointers to which templates may beused to synthesize the appropriate background.

We have made an early attempt with a talk-show video(from YouTube). Figure 9 shows how the backgroundcan be constructed by employing the “content awaredeletion” tool in Photoshop. Specifically, a human cropsthe foreground – the speaker in this video frame – andPhotoshop deletes this foreground to generate the frameon the right. This right frame is now carefully insertedto subsequent frames. Figure 10 shows the outcome– the left side frames are the originals, and the corre-sponding right side ones are synthesized. We foundthat these synthesized frames are quite viewable. Weasked 6 people to rate the quality of synthesized images.The average rating was 4 out of 5, demonstrating thatthe user’s satisfaction does not degrade excessively. Ofcourse, this is only a toy test – extensive experimentationis necessary to quantify the deeper tradeoffs.

Figure 9: (a) The original frame from a talk show. (b)Extracting the background.

4. LIMITATION AND FUTURE WORKThis section discusses some additional research ques-tions that would require future research attention.

Figure 10: Synthesizing backgrounds for videos.

Template backgrounds. We assumed that the back-ground of the original picture can be concisely summa-rized (e.g., cloudy blue sky). This may require sophisti-cated image processing, such as Google Goggles, or someform of crowd-sourcing to label pictures reasonably well.These kind of techniques are gaining traction [7], butmay not be fully mature in the near future.

Template substitution. Even if an appropriate back-ground template is available, we assumed that it can beinserted in a way that does not affect the foreground.For instance, when a Ferrari car is the foreground, andthe background is a view of a street, it is important tosuperimpose the Ferrari in a way that is meaningful.If the image operation inserts the Ferrari on the side-walk, or on top of other cars in the street, the imagewill be distorted. Focused research would be necessaryto synthesize the picture with respect to placement andproportion. Some image processing techniques are al-ready making progress in this direction [8]. Admittedly,insertion of a visually satisfying background without anyhuman assistance is a difficult task for computer vision.We hope content providers may have the incentive to in-vest effort in assisting create synthesized backgrounds.Moreover, under certain circumstances (e.g., extremelybad connection or very limited quota), another morestraightforward solution is to transmit the foregroundonly and delay the transmission of the background. Thiswill also leverage the need for template substitution.

Potential gain for video compression. Advanced videocompression techniques such as MPEG-7 [9] have al-ready eliminated most of information redundancy byexploiting similarity between successive frames. So anatural concern is what is the gain of our technique com-pared with these conventional techniques. Our view-point is that our technique can always be applied on topof any of these conventional technique since even thebackground of the key frames can be saved in our case.Of course, the fundamental tradeoff between bandwidth

5

and video quality will affect the overall gain. A morethorough analysis is one of the future work

Potential applications and requirements. Besides sav-ing bandwidth, context-aware compression can also beused towards variable-fidelity storage and informationdistillation. One may envision a surveillance camerarecording videos in variable fidelity (backgrounds ofolder videos proportionally reduced in fidelity). Futureresearch will need to explore the variety of applicationsin context-aware compression.

5. RELATED WORKThe idea of context-aware compression draws from mul-tiple threads of research, including compression, imageprocessing, and application-awareness.

Lossy Compression. Lossy compression [10] dedicatesto search for encodingds that can compress file sizesignificantly while Lossy compression [10] pertains toencoding algorithms that tradeoff information loss forreduced content volume. Although a mature field, thereis renewed interest here in light of the pressing needto reduce content size. Very recently, Google developedan image compression format “WebP” [11] that reducescontent size without affecting the viewing experiencetoo much. Authors in [12,13] have also looked into hu-man factors. They have observed that human attentionis usually drawn to certain visual features, and hence,images can be rendered with more details to such ob-jects. In contrast, our proposal is to exploit the humaninterest at the higher, semantic level – we extract andpreserve the main context as is.

Image processing. Our proposal relies heavily on im-age processing (especially object recognition) for taskssuch as selecting foreground/background, constructingand replacing backgrounds with templates. Current ef-fort in this field has made significant progress in iden-tifying objects [6], recognizing the characteristics [14],and even tagging them [15]. Further advancements willonly facilitate context-aware compression.

Application-awareness. The notion of application-awareness is broad and has been employed in vari-ous domains, such as operating systems and imagesearch [16, 17]. Authors in [16] introduced the notionof fidelity and the value of lowering fidelity for band-width savings. A related paper [18] showed the benefitsof application-awareness in energy saving. This paperdraws from a variety of these ideas.

6. CONCLUSIONThis paper proposes the notion of context-aware com-pression. The key observation is that sizable portions ofhuman-consumed content are not critical towards pre-

serving the semantic value of the content. One mayleverage this slack by transmitting only the contextuallyrelevant portions – the foreground – in full fidelity, andconcisely summarizing the background. Upon receivingthe information, the receiver may be able to approxi-mate the original content by “stitching” the foregroundwith a similar template background, drawn from its localcache. Such an approximation may be a “smaller priceto pay” in comparison to the rising cost of wireless band-width. Thus, although our ideas and results in this paperare preliminary, we find evidence to believe that context-aware compression can be a promising tool for a varietyof future applications.

7. REFERENCES[1] Rysavy Research, “Mobile Broadband Capacity

Constraints And the Need for Optimization,” 2010.[2] Nokia Siemens Networks, “Unite: Trends and insights

2009,” 2009.[3] A. Balasubramanian, R. Mahajan, and

A. Venkataramani, “Augmenting Mobile 3G Using WiFi,”in ACM Mobisys, 2010.

[4] J.H. Yun and K.G. Shin, “CTRL: A Self-OrganizingFemtocell Management Architecture for Co-ChannelDeployment,” ACM MobiCom, 2010.

[5] A.A. Sani, L. Zhong, and A. Sabharwal, “DirectionalAntenna Diversity for Mobile Devices: Characterizationsand Solutions,” in ACM MobiCom, 2010.

[6] B. Alexe, T. Deselaers, and V. Ferrari, “What is anobject,” CVPR, 2010.

[7] T. Yan, V. Kumar, and D. Ganesan, “CrowdSearch:exploiting crowds for accurate real-time image search onmobile phones,” in ACM MobiSys, 2010.

[8] P. Debevec, “Rendering synthetic objects into realscenes: Bridging traditional and image-based graphicswith global illumination and high dynamic rangephotography,” in ACM SIGGRAPH, 2008.

[9] ISO, “MPEG-7 Overview,” http: // mpeg.chiariglione. org/ standards/ mpeg-7/ mpeg-7. htm .

[10] G.K. Wallace, “The JPEG still picture compressionstandard,” IEEE TCE, 2002.

[11] Google, “Introduction to WebP on Google code,”http: // code. google. com/ speed/ webp/ .

[12] C. OSullivan and et. al., “Perceptually adaptivegraphics,” Eurographics State of the Art Reports, 2004.

[13] Seung-Hyun Lee, Sang-Bok Choi, and et. al.,“Non-uniform image compression using a biologicallymotivated selective attention model,” Neurocomputing.

[14] D.G. Lowe, “Object recognition from localscale-invariant features,” in ICCV, 1999.

[15] S. Belongie, J. Malik, and J. Puzicha, “Shape matchingand object recognition using shape contexts,” IEEETPAMI, 2002.

[16] B.D. Noble, M. Satyanarayanan, D. Narayanan, J.E.Tilton, J. Flinn, and K.R. Walker, “Agileapplication-aware adaptation for mobility,” ACM SOSP,1997.

[17] M. Satyanarayanan, R. Sukthankar, L. Mummert,A. Goode, J. Harkes, and S. Schlosser, “The uniquestrengths and storage access characteristics ofdiscard-based search,” Journal of Internet Services andApplications, vol. 1(1), 2010.

[18] J. Flinn and M. Satyanarayanan, “Energy-awareadaptation for mobile applications,” in ACM SOSP, 1999.

6