CEC: Continuous Eventual Checkpointing for Data Stream Processing
App Engine - fit.mta.edu.vnfit.mta.edu.vn/files/DanhSach/L11_App Engine_datastore.pdf · > All...
Transcript of App Engine - fit.mta.edu.vnfit.mta.edu.vn/files/DanhSach/L11_App Engine_datastore.pdf · > All...
![Page 1: App Engine - fit.mta.edu.vnfit.mta.edu.vn/files/DanhSach/L11_App Engine_datastore.pdf · > All server returns the same results. Eventual Takes time until all data becomes Consistency](https://reader033.fdocuments.us/reader033/viewer/2022050113/5f4aac69c514e7248e766e6c/html5/thumbnails/1.jpg)
App Engine: Datastore Introduction
Part 1
Another very useful course:https://www.udacity.com/course/developing-scalable-apps-in-java--ud859
1
![Page 2: App Engine - fit.mta.edu.vnfit.mta.edu.vn/files/DanhSach/L11_App Engine_datastore.pdf · > All server returns the same results. Eventual Takes time until all data becomes Consistency](https://reader033.fdocuments.us/reader033/viewer/2022050113/5f4aac69c514e7248e766e6c/html5/thumbnails/2.jpg)
Topics cover in this lessonTopics cover in this lesson
• What is Datastore?What is Datastore?– Datastore and relational database– Scalability, reliability and performancey, y p
• Datastore Internals– BigtableBigtable
• Datastore Basics Operation– Entity Properties and KeysEntity, Properties and Keys– Properties and Value Types– Datastore APIsDatastore APIs
2
![Page 3: App Engine - fit.mta.edu.vnfit.mta.edu.vn/files/DanhSach/L11_App Engine_datastore.pdf · > All server returns the same results. Eventual Takes time until all data becomes Consistency](https://reader033.fdocuments.us/reader033/viewer/2022050113/5f4aac69c514e7248e766e6c/html5/thumbnails/3.jpg)
What is Datastore?What is Datastore?
• Datastore is a database (persistent storage)Datastore is a database (persistent storage) for AppEngine
AppEngine Traditional Web Apps
Web application
AppEngine(Java, Python,
Pert/CGIPHP
framework PHP, Go) Ruby on RailsPersistent storage
Datastore RDBMS: MySQL MSstorage MySQL, MS SQL, Oracle
3
![Page 4: App Engine - fit.mta.edu.vnfit.mta.edu.vn/files/DanhSach/L11_App Engine_datastore.pdf · > All server returns the same results. Eventual Takes time until all data becomes Consistency](https://reader033.fdocuments.us/reader033/viewer/2022050113/5f4aac69c514e7248e766e6c/html5/thumbnails/4.jpg)
What is Datastore?
• Persistent storage for AppEngine
What is Datastore?
Persistent storage for AppEngine• AppEngine is very scalable=> Many instances
=> Central Server to store data from all=> Central Server to store data from all instances.Wh RDB? S l bili !• Why not RDB? Scalability!
4
![Page 5: App Engine - fit.mta.edu.vnfit.mta.edu.vn/files/DanhSach/L11_App Engine_datastore.pdf · > All server returns the same results. Eventual Takes time until all data becomes Consistency](https://reader033.fdocuments.us/reader033/viewer/2022050113/5f4aac69c514e7248e766e6c/html5/thumbnails/5.jpg)
Datastore and RDBMSDatastore and RDBMS
Datastore RDBMSDatastore RDBMS
Query SQL like query Full support of SQLQuery language flexibility
SQL-like query language : Limited to simple filter and
Full support of SQL-Table JOIN- Flexible filtering
sort - SubqueryReliability and S l bilit
Highly scalable and li bl ith
Hard to scaleScalability reliable with
performance
D t t ff G l l l l bilit
5
Datastore offsers Google-level scalability
![Page 6: App Engine - fit.mta.edu.vnfit.mta.edu.vn/files/DanhSach/L11_App Engine_datastore.pdf · > All server returns the same results. Eventual Takes time until all data becomes Consistency](https://reader033.fdocuments.us/reader033/viewer/2022050113/5f4aac69c514e7248e766e6c/html5/thumbnails/6.jpg)
Problems of Scalability and ReliabilityProblems of Scalability and Reliability
• Single InstanceSingle Instance– Performance limited by machine resource
Single point of failure– Single point of failure
• Replication (copies) increases reliability– Consistency among instances
• Sharding (Split among machines)– Lock control (transaction)[Shard = split server into multiple machines]
6
![Page 7: App Engine - fit.mta.edu.vnfit.mta.edu.vn/files/DanhSach/L11_App Engine_datastore.pdf · > All server returns the same results. Eventual Takes time until all data becomes Consistency](https://reader033.fdocuments.us/reader033/viewer/2022050113/5f4aac69c514e7248e766e6c/html5/thumbnails/7.jpg)
Strong Consistency and Eventual Consistency
Strong Data is always consistent among allStrong Consistency
Data is always consistent among all database instances-Just after write operation- Crash in the middle of write operation -> All server returns the same results.
Eventual Takes time until all data becomesEventual Consistency
Takes time until all data becomesconsistent after write(Think of DNS as an example)
DNS i di t ib t d d t b tDNS is a distributed database system.Updated configuration on domain -> Not reflected to all DNS immediately. For a certain period of time, some DNS servers return old. 7
![Page 8: App Engine - fit.mta.edu.vnfit.mta.edu.vn/files/DanhSach/L11_App Engine_datastore.pdf · > All server returns the same results. Eventual Takes time until all data becomes Consistency](https://reader033.fdocuments.us/reader033/viewer/2022050113/5f4aac69c514e7248e766e6c/html5/thumbnails/8.jpg)
Scalability, Reliability and Performance on RDB
• Replication and/or sharding for scalabilityReplication and/or sharding for scalability• But…
St i t RDB l it ti– Strong consistency on RDB slows write operations due to lock.Join operation is a bottleneck due to data– Join operation is a bottleneck due to data shuffling.
RDB ensures strong consistency -> Hard to ensure scalability.
8Datastore for AppEngine
![Page 9: App Engine - fit.mta.edu.vnfit.mta.edu.vn/files/DanhSach/L11_App Engine_datastore.pdf · > All server returns the same results. Eventual Takes time until all data becomes Consistency](https://reader033.fdocuments.us/reader033/viewer/2022050113/5f4aac69c514e7248e766e6c/html5/thumbnails/9.jpg)
Datastore InternalsDatastore Internals• Based on Bigtable, which offers super high
l biliscalability.• High availability by High Replication Datastore
(HRD)– Synchronous write on multiple datacenters.
• Supports strong consistency among multiple rows
9
![Page 10: App Engine - fit.mta.edu.vnfit.mta.edu.vn/files/DanhSach/L11_App Engine_datastore.pdf · > All server returns the same results. Eventual Takes time until all data becomes Consistency](https://reader033.fdocuments.us/reader033/viewer/2022050113/5f4aac69c514e7248e766e6c/html5/thumbnails/10.jpg)
What is Bigtable?What is Bigtable?
• Scalable, distributed, highly-available and Sca ab e, d st buted, g y a a ab e a dstructured storage– Bigtable is not database itself (it doesn’t support
query)• Consistency
S i f i l– Strong consistency for single row– Eventual consistency for multi-row level
• Google usage• Google usage– In production since April 2005– Web search youtubeWeb search, youtube…
10
![Page 11: App Engine - fit.mta.edu.vnfit.mta.edu.vn/files/DanhSach/L11_App Engine_datastore.pdf · > All server returns the same results. Eventual Takes time until all data becomes Consistency](https://reader033.fdocuments.us/reader033/viewer/2022050113/5f4aac69c514e7248e766e6c/html5/thumbnails/11.jpg)
Automatic Scale-out of Bigtable table server
11
![Page 12: App Engine - fit.mta.edu.vnfit.mta.edu.vn/files/DanhSach/L11_App Engine_datastore.pdf · > All server returns the same results. Eventual Takes time until all data becomes Consistency](https://reader033.fdocuments.us/reader033/viewer/2022050113/5f4aac69c514e7248e766e6c/html5/thumbnails/12.jpg)
Bigtable Data ModelBigtable Data Model
• Key value data storageKey value data storage• A row has a Key and Columns
S d b• Sorted by Key– In lexical order– Enables range query by application
12
![Page 13: App Engine - fit.mta.edu.vnfit.mta.edu.vn/files/DanhSach/L11_App Engine_datastore.pdf · > All server returns the same results. Eventual Takes time until all data becomes Consistency](https://reader033.fdocuments.us/reader033/viewer/2022050113/5f4aac69c514e7248e766e6c/html5/thumbnails/13.jpg)
Bigtable OperationsBigtable Operations
• CRUD on a rowCRUD on a row– Create, Read, Update and Delete operations
Preserves single row strong consistency (not– Preserves single-row strong consistency (not multiple row).
• Scan by range of keys• Scan by range of keys– But can not search by column values
13
![Page 14: App Engine - fit.mta.edu.vnfit.mta.edu.vn/files/DanhSach/L11_App Engine_datastore.pdf · > All server returns the same results. Eventual Takes time until all data becomes Consistency](https://reader033.fdocuments.us/reader033/viewer/2022050113/5f4aac69c514e7248e766e6c/html5/thumbnails/14.jpg)
Scalability is based on Bigtable automated sharding
14
Scalability is based on Bigtable automated sharding.Megastore supports transactions (strong consistency)
![Page 15: App Engine - fit.mta.edu.vnfit.mta.edu.vn/files/DanhSach/L11_App Engine_datastore.pdf · > All server returns the same results. Eventual Takes time until all data becomes Consistency](https://reader033.fdocuments.us/reader033/viewer/2022050113/5f4aac69c514e7248e766e6c/html5/thumbnails/15.jpg)
15Property = actual data you want to store
![Page 16: App Engine - fit.mta.edu.vnfit.mta.edu.vn/files/DanhSach/L11_App Engine_datastore.pdf · > All server returns the same results. Eventual Takes time until all data becomes Consistency](https://reader033.fdocuments.us/reader033/viewer/2022050113/5f4aac69c514e7248e766e6c/html5/thumbnails/16.jpg)
16
![Page 17: App Engine - fit.mta.edu.vnfit.mta.edu.vn/files/DanhSach/L11_App Engine_datastore.pdf · > All server returns the same results. Eventual Takes time until all data becomes Consistency](https://reader033.fdocuments.us/reader033/viewer/2022050113/5f4aac69c514e7248e766e6c/html5/thumbnails/17.jpg)
17Property can have multiple values. (Multiple data for one property)
![Page 18: App Engine - fit.mta.edu.vnfit.mta.edu.vn/files/DanhSach/L11_App Engine_datastore.pdf · > All server returns the same results. Eventual Takes time until all data becomes Consistency](https://reader033.fdocuments.us/reader033/viewer/2022050113/5f4aac69c514e7248e766e6c/html5/thumbnails/18.jpg)
18
![Page 19: App Engine - fit.mta.edu.vnfit.mta.edu.vn/files/DanhSach/L11_App Engine_datastore.pdf · > All server returns the same results. Eventual Takes time until all data becomes Consistency](https://reader033.fdocuments.us/reader033/viewer/2022050113/5f4aac69c514e7248e766e6c/html5/thumbnails/19.jpg)
19
![Page 20: App Engine - fit.mta.edu.vnfit.mta.edu.vn/files/DanhSach/L11_App Engine_datastore.pdf · > All server returns the same results. Eventual Takes time until all data becomes Consistency](https://reader033.fdocuments.us/reader033/viewer/2022050113/5f4aac69c514e7248e766e6c/html5/thumbnails/20.jpg)
20
![Page 21: App Engine - fit.mta.edu.vnfit.mta.edu.vn/files/DanhSach/L11_App Engine_datastore.pdf · > All server returns the same results. Eventual Takes time until all data becomes Consistency](https://reader033.fdocuments.us/reader033/viewer/2022050113/5f4aac69c514e7248e766e6c/html5/thumbnails/21.jpg)
App Engine: Datastore Query, Index andDatastore Query, Index and
Transactions
Part 2
21
![Page 22: App Engine - fit.mta.edu.vnfit.mta.edu.vn/files/DanhSach/L11_App Engine_datastore.pdf · > All server returns the same results. Eventual Takes time until all data becomes Consistency](https://reader033.fdocuments.us/reader033/viewer/2022050113/5f4aac69c514e7248e766e6c/html5/thumbnails/22.jpg)
2222
![Page 23: App Engine - fit.mta.edu.vnfit.mta.edu.vn/files/DanhSach/L11_App Engine_datastore.pdf · > All server returns the same results. Eventual Takes time until all data becomes Consistency](https://reader033.fdocuments.us/reader033/viewer/2022050113/5f4aac69c514e7248e766e6c/html5/thumbnails/23.jpg)
2323
![Page 24: App Engine - fit.mta.edu.vnfit.mta.edu.vn/files/DanhSach/L11_App Engine_datastore.pdf · > All server returns the same results. Eventual Takes time until all data becomes Consistency](https://reader033.fdocuments.us/reader033/viewer/2022050113/5f4aac69c514e7248e766e6c/html5/thumbnails/24.jpg)
2424
![Page 25: App Engine - fit.mta.edu.vnfit.mta.edu.vn/files/DanhSach/L11_App Engine_datastore.pdf · > All server returns the same results. Eventual Takes time until all data becomes Consistency](https://reader033.fdocuments.us/reader033/viewer/2022050113/5f4aac69c514e7248e766e6c/html5/thumbnails/25.jpg)
2525
![Page 26: App Engine - fit.mta.edu.vnfit.mta.edu.vn/files/DanhSach/L11_App Engine_datastore.pdf · > All server returns the same results. Eventual Takes time until all data becomes Consistency](https://reader033.fdocuments.us/reader033/viewer/2022050113/5f4aac69c514e7248e766e6c/html5/thumbnails/26.jpg)
Bigtable can scanon a key, noton a key, not value!
Index table on Bigtable:Property name and valueImplement query on bigtable(without reading
t l l )actual value)
2626
![Page 27: App Engine - fit.mta.edu.vnfit.mta.edu.vn/files/DanhSach/L11_App Engine_datastore.pdf · > All server returns the same results. Eventual Takes time until all data becomes Consistency](https://reader033.fdocuments.us/reader033/viewer/2022050113/5f4aac69c514e7248e766e6c/html5/thumbnails/27.jpg)
2727
![Page 28: App Engine - fit.mta.edu.vnfit.mta.edu.vn/files/DanhSach/L11_App Engine_datastore.pdf · > All server returns the same results. Eventual Takes time until all data becomes Consistency](https://reader033.fdocuments.us/reader033/viewer/2022050113/5f4aac69c514e7248e766e6c/html5/thumbnails/28.jpg)
2828
![Page 29: App Engine - fit.mta.edu.vnfit.mta.edu.vn/files/DanhSach/L11_App Engine_datastore.pdf · > All server returns the same results. Eventual Takes time until all data becomes Consistency](https://reader033.fdocuments.us/reader033/viewer/2022050113/5f4aac69c514e7248e766e6c/html5/thumbnails/29.jpg)
2929
![Page 30: App Engine - fit.mta.edu.vnfit.mta.edu.vn/files/DanhSach/L11_App Engine_datastore.pdf · > All server returns the same results. Eventual Takes time until all data becomes Consistency](https://reader033.fdocuments.us/reader033/viewer/2022050113/5f4aac69c514e7248e766e6c/html5/thumbnails/30.jpg)
3030
![Page 31: App Engine - fit.mta.edu.vnfit.mta.edu.vn/files/DanhSach/L11_App Engine_datastore.pdf · > All server returns the same results. Eventual Takes time until all data becomes Consistency](https://reader033.fdocuments.us/reader033/viewer/2022050113/5f4aac69c514e7248e766e6c/html5/thumbnails/31.jpg)
3131
![Page 32: App Engine - fit.mta.edu.vnfit.mta.edu.vn/files/DanhSach/L11_App Engine_datastore.pdf · > All server returns the same results. Eventual Takes time until all data becomes Consistency](https://reader033.fdocuments.us/reader033/viewer/2022050113/5f4aac69c514e7248e766e6c/html5/thumbnails/32.jpg)
3232
![Page 33: App Engine - fit.mta.edu.vnfit.mta.edu.vn/files/DanhSach/L11_App Engine_datastore.pdf · > All server returns the same results. Eventual Takes time until all data becomes Consistency](https://reader033.fdocuments.us/reader033/viewer/2022050113/5f4aac69c514e7248e766e6c/html5/thumbnails/33.jpg)
3333
![Page 34: App Engine - fit.mta.edu.vnfit.mta.edu.vn/files/DanhSach/L11_App Engine_datastore.pdf · > All server returns the same results. Eventual Takes time until all data becomes Consistency](https://reader033.fdocuments.us/reader033/viewer/2022050113/5f4aac69c514e7248e766e6c/html5/thumbnails/34.jpg)
3434
![Page 35: App Engine - fit.mta.edu.vnfit.mta.edu.vn/files/DanhSach/L11_App Engine_datastore.pdf · > All server returns the same results. Eventual Takes time until all data becomes Consistency](https://reader033.fdocuments.us/reader033/viewer/2022050113/5f4aac69c514e7248e766e6c/html5/thumbnails/35.jpg)
3535
![Page 36: App Engine - fit.mta.edu.vnfit.mta.edu.vn/files/DanhSach/L11_App Engine_datastore.pdf · > All server returns the same results. Eventual Takes time until all data becomes Consistency](https://reader033.fdocuments.us/reader033/viewer/2022050113/5f4aac69c514e7248e766e6c/html5/thumbnails/36.jpg)
3636
![Page 37: App Engine - fit.mta.edu.vnfit.mta.edu.vn/files/DanhSach/L11_App Engine_datastore.pdf · > All server returns the same results. Eventual Takes time until all data becomes Consistency](https://reader033.fdocuments.us/reader033/viewer/2022050113/5f4aac69c514e7248e766e6c/html5/thumbnails/37.jpg)
3737
![Page 38: App Engine - fit.mta.edu.vnfit.mta.edu.vn/files/DanhSach/L11_App Engine_datastore.pdf · > All server returns the same results. Eventual Takes time until all data becomes Consistency](https://reader033.fdocuments.us/reader033/viewer/2022050113/5f4aac69c514e7248e766e6c/html5/thumbnails/38.jpg)
3838
![Page 39: App Engine - fit.mta.edu.vnfit.mta.edu.vn/files/DanhSach/L11_App Engine_datastore.pdf · > All server returns the same results. Eventual Takes time until all data becomes Consistency](https://reader033.fdocuments.us/reader033/viewer/2022050113/5f4aac69c514e7248e766e6c/html5/thumbnails/39.jpg)
3939
![Page 40: App Engine - fit.mta.edu.vnfit.mta.edu.vn/files/DanhSach/L11_App Engine_datastore.pdf · > All server returns the same results. Eventual Takes time until all data becomes Consistency](https://reader033.fdocuments.us/reader033/viewer/2022050113/5f4aac69c514e7248e766e6c/html5/thumbnails/40.jpg)
4040
![Page 41: App Engine - fit.mta.edu.vnfit.mta.edu.vn/files/DanhSach/L11_App Engine_datastore.pdf · > All server returns the same results. Eventual Takes time until all data becomes Consistency](https://reader033.fdocuments.us/reader033/viewer/2022050113/5f4aac69c514e7248e766e6c/html5/thumbnails/41.jpg)
4141
![Page 42: App Engine - fit.mta.edu.vnfit.mta.edu.vn/files/DanhSach/L11_App Engine_datastore.pdf · > All server returns the same results. Eventual Takes time until all data becomes Consistency](https://reader033.fdocuments.us/reader033/viewer/2022050113/5f4aac69c514e7248e766e6c/html5/thumbnails/42.jpg)
4242
![Page 43: App Engine - fit.mta.edu.vnfit.mta.edu.vn/files/DanhSach/L11_App Engine_datastore.pdf · > All server returns the same results. Eventual Takes time until all data becomes Consistency](https://reader033.fdocuments.us/reader033/viewer/2022050113/5f4aac69c514e7248e766e6c/html5/thumbnails/43.jpg)
4343
![Page 44: App Engine - fit.mta.edu.vnfit.mta.edu.vn/files/DanhSach/L11_App Engine_datastore.pdf · > All server returns the same results. Eventual Takes time until all data becomes Consistency](https://reader033.fdocuments.us/reader033/viewer/2022050113/5f4aac69c514e7248e766e6c/html5/thumbnails/44.jpg)
4444
![Page 45: App Engine - fit.mta.edu.vnfit.mta.edu.vn/files/DanhSach/L11_App Engine_datastore.pdf · > All server returns the same results. Eventual Takes time until all data becomes Consistency](https://reader033.fdocuments.us/reader033/viewer/2022050113/5f4aac69c514e7248e766e6c/html5/thumbnails/45.jpg)
4545
![Page 46: App Engine - fit.mta.edu.vnfit.mta.edu.vn/files/DanhSach/L11_App Engine_datastore.pdf · > All server returns the same results. Eventual Takes time until all data becomes Consistency](https://reader033.fdocuments.us/reader033/viewer/2022050113/5f4aac69c514e7248e766e6c/html5/thumbnails/46.jpg)
4646
![Page 47: App Engine - fit.mta.edu.vnfit.mta.edu.vn/files/DanhSach/L11_App Engine_datastore.pdf · > All server returns the same results. Eventual Takes time until all data becomes Consistency](https://reader033.fdocuments.us/reader033/viewer/2022050113/5f4aac69c514e7248e766e6c/html5/thumbnails/47.jpg)
4747
![Page 48: App Engine - fit.mta.edu.vnfit.mta.edu.vn/files/DanhSach/L11_App Engine_datastore.pdf · > All server returns the same results. Eventual Takes time until all data becomes Consistency](https://reader033.fdocuments.us/reader033/viewer/2022050113/5f4aac69c514e7248e766e6c/html5/thumbnails/48.jpg)
4848
Caveats = limitations
![Page 49: App Engine - fit.mta.edu.vnfit.mta.edu.vn/files/DanhSach/L11_App Engine_datastore.pdf · > All server returns the same results. Eventual Takes time until all data becomes Consistency](https://reader033.fdocuments.us/reader033/viewer/2022050113/5f4aac69c514e7248e766e6c/html5/thumbnails/49.jpg)
4949
![Page 50: App Engine - fit.mta.edu.vnfit.mta.edu.vn/files/DanhSach/L11_App Engine_datastore.pdf · > All server returns the same results. Eventual Takes time until all data becomes Consistency](https://reader033.fdocuments.us/reader033/viewer/2022050113/5f4aac69c514e7248e766e6c/html5/thumbnails/50.jpg)
5050
![Page 51: App Engine - fit.mta.edu.vnfit.mta.edu.vn/files/DanhSach/L11_App Engine_datastore.pdf · > All server returns the same results. Eventual Takes time until all data becomes Consistency](https://reader033.fdocuments.us/reader033/viewer/2022050113/5f4aac69c514e7248e766e6c/html5/thumbnails/51.jpg)
5151
![Page 52: App Engine - fit.mta.edu.vnfit.mta.edu.vn/files/DanhSach/L11_App Engine_datastore.pdf · > All server returns the same results. Eventual Takes time until all data becomes Consistency](https://reader033.fdocuments.us/reader033/viewer/2022050113/5f4aac69c514e7248e766e6c/html5/thumbnails/52.jpg)
5252
![Page 53: App Engine - fit.mta.edu.vnfit.mta.edu.vn/files/DanhSach/L11_App Engine_datastore.pdf · > All server returns the same results. Eventual Takes time until all data becomes Consistency](https://reader033.fdocuments.us/reader033/viewer/2022050113/5f4aac69c514e7248e766e6c/html5/thumbnails/53.jpg)
5353
![Page 54: App Engine - fit.mta.edu.vnfit.mta.edu.vn/files/DanhSach/L11_App Engine_datastore.pdf · > All server returns the same results. Eventual Takes time until all data becomes Consistency](https://reader033.fdocuments.us/reader033/viewer/2022050113/5f4aac69c514e7248e766e6c/html5/thumbnails/54.jpg)
5454
![Page 55: App Engine - fit.mta.edu.vnfit.mta.edu.vn/files/DanhSach/L11_App Engine_datastore.pdf · > All server returns the same results. Eventual Takes time until all data becomes Consistency](https://reader033.fdocuments.us/reader033/viewer/2022050113/5f4aac69c514e7248e766e6c/html5/thumbnails/55.jpg)
5555
![Page 56: App Engine - fit.mta.edu.vnfit.mta.edu.vn/files/DanhSach/L11_App Engine_datastore.pdf · > All server returns the same results. Eventual Takes time until all data becomes Consistency](https://reader033.fdocuments.us/reader033/viewer/2022050113/5f4aac69c514e7248e766e6c/html5/thumbnails/56.jpg)
5656
![Page 57: App Engine - fit.mta.edu.vnfit.mta.edu.vn/files/DanhSach/L11_App Engine_datastore.pdf · > All server returns the same results. Eventual Takes time until all data becomes Consistency](https://reader033.fdocuments.us/reader033/viewer/2022050113/5f4aac69c514e7248e766e6c/html5/thumbnails/57.jpg)
57