OS Project 2011 Documentation
-
Upload
gamal-magdy -
Category
Documents
-
view
216 -
download
0
Transcript of OS Project 2011 Documentation
-
8/6/2019 OS Project 2011 Documentation
1/29
Faculty of Computer and Information Sciences
Ain Shams University
Third Year Second Semester
2010 - 2011
Dr. Howaida Abdel Al Fattah
-
8/6/2019 OS Project 2011 Documentation
2/29
2
Contents
INTRODUCTION .......................................................................................................................... 4Whats new?! ........................................................................................................................................................4Project Specifications .............................................................................................................................................. 5New Concepts........................................................................................................................................................6
The Working Sets ................................................................................................................................................ 6The Page File ...................................................................................................................................................... 6Page Buffering .................................................................................................................................................... 7
OVERALL VIEW........................................................................................................................... 8DETAILS ..................................................................................................................................... 9env_create( )
.........................................................................................................................................................
9Fault Handler .........................................................................................................................................................9env_free( ) .......................................................................................................................................................... 13Dynamic allocation/free ........................................................................................................................................ 13BONUSES .................................................................................................................................. 15First: Freeing RAM when it's Full ............................................................................................................................. 15Second: Buffering Page Tables ................................................................................................................................ 15CHALLENGE!!............................................................................................................................ 15HOW TO TEST YOUR PROJECT .................................................................................................. 16FIRST: Testing Each Part......................................................................................................................................... 16
Testing Page & Table Placement:.......................................................................................................................... 16Testing Page Replacement: ................................................................................................................................. 16Testing Table Replacement:
................................................................................................................................
16Testing Page & Table Replacement: ...................................................................................................................... 16Testing Dynamic Allocation & Free: ...................................................................................................................... 17
SECOND: Testing Whole Project .............................................................................................................................. 17Scenario 1: Env_Free.......................................................................................................................................... 17Scenario 2: Small Programs ................................................................................................................................. 17Scenario 3: QuickSort Program ............................................................................................................................ 17Scenario 4: QuickSort_freeHeap Program .............................................................................................................. 18
APPENDIX I: PAGE FILE HELPER FUNCTIONS............................................................................
19Add a new environment page to the page file ........................................................................................................... 19
-
8/6/2019 OS Project 2011 Documentation
3/29
3
Read an environment page from the page file to the main memory.............................................................................. 19Update certain environment page in the page file by contents from the main memory ...................................................20Read an environment table from the page file to the main memory ............................................................................. 20Write a certain environment table to the page file..................................................................................................... 21Remove an existing environment page/table from the page file
..................................................................................
21APPENDIX II: WORKING SET STRUCTURE & HELPER FUNCTIONS............................................. 22Working Set Structure ........................................................................................................................................... 22Working Set Functions .......................................................................................................................................... 22
inline uint32 env_page_ws_get_size(struct Env *e) ................................................................................................. 22inline uint32 env_table_ws_get_size(struct Env *e) ................................................................................................. 22inline void env_page_ws_invalidate(struct Env* e, uint32 virtual_address) .................................................................. 23inline void env_table_ws_invalidate(struct Env* e, uint32 virtual_address) .................................................................. 23inline void env_page_ws_set_entry(struct Env* e, uint32 entry_index, uint32 virtual_address)
.......................................
23inline void env_table_ws_set_entry(struct Env* e, uint32 entry_index, uint32 virtual_address) ....................................... 23inline void env_page_ws_clear_entry(struct Env* e, uint32 entry_index)..................................................................... 23inline void env_table_ws_clear_entry(struct Env* e, uint32 entry_index) .................................................................... 23inline uint32 env_page_ws_get_virtual_address(struct Env* e, uint32 entry_index) ......................................................23inline uint32 env_table_ws_get_virtual_address(struct Env* e, uint32 entry_index) ......................................................23inline uint32 env_page_ws_is_entry_empty(struct Env* e, uint32 entry_index) ............................................................ 24inline uint32 env_table_ws_is_entry_empty(struct Env* e, uint32 entry_index) ...........................................................24inline voidenv_page_ws_print(struct Env* e)........................................................................................................ 24inline voidenv_table_ws_print(struct Env* e) ....................................................................................................... 24
APPENDIX III: FUNCTIONS FOR MANIPULATING PERMISSIONS IN PAGE TABLES AND
DIRECTORY.............................................................................................................................. 25inline void pt_set_page_permissions(struct Env* ptr_env, uint32 virtual_address, uint32 permissions_to_set, uint32
permissions_to_clear) ........................................................................................................................................ 25inline uint32 pt_get_page_permissions(struct Env* ptr_env, uint32 virtual_address ) ....................................................25inline uint32 pd_is_table_used(uint32* ptr_pgdir, uint32 virtual_address) .................................................................. 26inline void pd_set_table_unused(uint32* ptr_pgdir, uint32 virtual_address) ................................................................ 26
APPENDIX IV: PAGE BUFFERING ............................................................................................... 27Structures ........................................................................................................................................................... 27Functions ............................................................................................................................................................ 27
inline void bufferList_add_page(struct Linked_List* bufferList,struct Frame_Info *ptr_frame_info) .................................. 27inline void bufferlist_remove_page(struct Linked_List* bufferList, struct Frame_Info *ptr_frame_info) ............................ 27
APPENDIX V: BASIC AND HELPER MEMORY MANAGEMENT FUNCTIONS .................................. 28Basic Functions .................................................................................................................................................... 28Helpers Functions................................................................................................................................................. 28
-
8/6/2019 OS Project 2011 Documentation
4/29
4
Introduction
Whats new?!
Previously:all segments of the program binary plus the stack page should be loaded in the main memory. If theres
no enough memory, the program will not be loaded. See the following figure:
But, wait a minute
This means that you may not be able to run one of your programs if theres no enough main memory for it!! This is
not the case in real OS!! In windows for example, you can run any program you want regardless the size of main
memory do you know WHY?!
YES it uses part ofsecondary memory as a virtual memory. So, the loading of the program will be distributed
between the main memory and secondary memory.
NOW in the project: when loading a program, only part of it will be loaded in the main memory while the whole
program will be loaded on the secondary memory (H.D.D.). See the following figure:
This means that a Page Fault can occur during the program run. (i.e. an exception thrown by the processor to
indicate that a page is not present in main memory). The kernel should handle it by loading the faulted page back to
the main memory from the secondary memory (H.D.D.).
BINRequired Tables
All
Pages
Set of pages
env_create( )
BIN
env_create( )
All
Ifno enough memory, then
dont load it
Group Members: 3-6
Group Registration: you shouldregister your group heredue to FRI 20/5/2011
Due to Date: Week #13 starting from SUN29/5/2011 to THU 2/6/2011. (Subject to change according to lab exam
schedule)
https://spreadsheets1.google.com/spreadsheet/viewform?hl=en&hl=en&formkey=dEUtWC1lRXlsQUY3NGJJVHRDa29mREE6MA#gid=0https://spreadsheets1.google.com/spreadsheet/viewform?hl=en&hl=en&formkey=dEUtWC1lRXlsQUY3NGJJVHRDa29mREE6MA#gid=0https://spreadsheets1.google.com/spreadsheet/viewform?hl=en&hl=en&formkey=dEUtWC1lRXlsQUY3NGJJVHRDa29mREE6MA#gid=0https://spreadsheets1.google.com/spreadsheet/viewform?hl=en&hl=en&formkey=dEUtWC1lRXlsQUY3NGJJVHRDa29mREE6MA#gid=0 -
8/6/2019 OS Project 2011 Documentation
5/29
5
Project Specifications
In the light of the studied memory management system, you are required add new features for your FOS memory
manager, these new features are:
For loading only part of a program in main memory, we use the working set concept; the working set is the set of all
pages (or tables) loaded in main memory of the program at run time at any instant of time.
Each program environment is modified to hold its working sets information. The working sets are a FIXED size array
of virtual addresses corresponding to pages that are loaded in main memory.
The virtual space of any loaded user application is as described in lab 6, see figure 1.
User Stack
..
VirtualMemory
256 MB
4 GB
KERNEL_BASE
Free Memory
USER_LIMIT
USER_TOP
8 MBFOS stackKERNEL_STACK_TOP
FOS Kernel
FOS stack
< 1 MB
User Heap
Memory 2 GB
1.0 GB
User Code +Data
0
USER_HEAP_START
USER_HEAP_MAX
USTACKTOP
Figure 1: Virtual space layout of single user loaded program
There are two important concepts you need to understand in order to implement the project; these concepts are the
Working Set and Page File.
1- Load multiple user programsthat are partially loaded in main memory and fully loadedin secondary memory (H.D.D.)
2- Handle page faults during execution3- Exit any of the loaded programs: (i.e. remove it from the main and secondary memory)4- Allow user program to dynamically allocate and free memory space at run-time (i.e.
mallocandfreeMem functions)
-
8/6/2019 OS Project 2011 Documentation
6/29
6
New Concepts
The Working Sets
We have previously defined the working set as the set of all pages (or tables) loaded in main memory of the program
at run time which is implemented as a FIXED size array of virtual addresses corresponding to pages that are loaded in
main memory.
SeeAppendix IIfor description about the working sets data structure and helper functions.
The maximum allowed number of pages (or tables) for any running user program in memory is defined inside FOS1,
let's say Npages (or M tables), when the program needs memory (either when the program is being loaded or by
dynamic allocation during the program run) FOS must allow maximum Npages (or M tables) for the program in main
memory.
So, when page fault occur during the program execution, the page (or table) should be loaded from the secondary
memory (the PAGE FILE as described later) to the WORKING SET of the program. If the working set is complete, then
one of the program environment pages (or tables) from the working set should be replaced with the new page (or
table). This is called LOCAL replacement since each program should replace one of its own loaded pages (or tables).
This means that our working sets are FIXED size with LOCAL replacement facility.
FOS must maintain the working sets during the run time of the program, when new pages (or tables) from PAGE FILE
are loaded to main memory, the working set must be updated.
Therefore the working sets of any loaded program must always contain the correct information about the pages (or
tables) loaded in main memory.
The Page File
The page file is an area in the secondary memory (H.D.D.) that is used for saving and loading programs pages (or
tables) during runtime. Thus, for each running program, there is a storage space in the page file forALL pages
needed by the program, this means that user code, data, stack and heap sections areALL written in page file.
(Remember that not all these pages are in main memory, only the working set).
You might wonder why we need to keep all pages of the program in secondary memory during run!!
The reason for this is to have a copy of each page in the page file. So, we dont need to write back each swapped out
page to the page file. Only MODIFIED pages are written back to the page file.
But wait a minute
Did we have a file manager in FOS?!!
NO!
Dont panic, we wrote some helper functions for you that allow us to deal with the page file. These functions provide
the following facilities:
1 defined in inc/environment_definitions.h with name "PAGE_WS_MAX_SIZE" and "TABLE_WS_MAX_SIZE"
The initialization of the working set by the set of pages during the program loading is already implemented
for you in "env_create()", andyou will be responsible for maintaining the working set during the run time of
the program.
-
8/6/2019 OS Project 2011 Documentation
7/29
7
1- Add a new environment empty page to the page file.2- Read an environment page (or a table) from the page file to the main memory.3- Update certain environment page (or WRITE a table) in the page file from the main memory.4- Remove an existing environment page (or a table) from the page file.
SeeAppendix Ifor description about these helper functions.
Page Buffering
An interesting strategy that can improve paging performance is page buffering. A replaced page is not lost but rather
is assigned to one of two lists:
1. the free frame list if the page has not been modified, or2. the modified page list if it has.
Note that the page is not physically moved about in main memory; instead, the PRESENT bit for this page is set to 0
and its "Frame_Info" is placed in either the free or modified list.
When a page is to be read in, the frame at the head of the free frame list is used, destroying the page that was there.
When an unmodified page is to be replaced, it remains in memory and its "Frame_info" is added to the tail of thefree frame list. Similarly, when a modified page is to be written out and replaced, its "Frame_info" is added to the
tail of the modified page list.
The pages on the modified list can periodically be written out in batches when their count reaches
"MAX_MODIFIED_LIST_COUNT2" and moved to the unmodified list. A page on the unmodified list is either
reclaimed if it is referenced, or lost when its frame is assigned to another page.
There are two important aspects of these maneuvers
1. The page to be replaced remains in memory. Thus if the process references that page, it is returned to theworking set of that process at little cost (by setting its PRESENT bit to 1).
2. Modified pages are written out in clusters rather than one at a time. This significantly reduces the number ofI/O operations and therefore the amount of disk access time.
2 defined in inc/environment_definitions.h
The loading ofALL program binary segments plus the stack page to the page file is already implemented for
you in "env_create()", you will need to maintain the page file in the rest of the project.
-
8/6/2019 OS Project 2011 Documentation
8/29
8
Overall ViewThe following figure shows an overall view of all project components together with the interaction between them.
The components marked with (*) should be written by you. The unmarked components are already implemented.
Figure 2: Interaction among components in project
Exiting
(Unloading)
Handling Page Faults (Fault Handler) *Page Placement
NotBuffered
Dynamic Alloc/Free
Running
Fault when accessing page (or
table) not in main memory
* Remove WorkingSets of Pages and Page
Tables + Directory
Remove all pages
and tables (if any)
from page file
freeHeap( ) *malloc( ) *
Nothing
Created in
Memory
ALL Pages
Added to
Page File
Working Sets
Pages + Tables in
Given Range
ALL Pages
+ Tables
env_run()env_free()
Page Re-placement
Stack
Add Page
IfNew stack page then add it to page file
env_create( )
BINWorking Sets
All
Program
Loaded
Pages Working Set in Memory
Tables Working Set in Memory
All pages in Page File
No tables initially on Page File
Handling Table Faults (Fault Handler) *Table Placement Table Re-placement
NEW
Exist
NEW
Exist
malloc( ) / freeHeap( )
VictimBuffered f.f.l.
mod.l.
f.f.l.
mod.l.
Not
Buffered
Buffered f.f.l.mod.l.
Write Pages
IfModified Buffer reaches Max then write it into page file
mod.l.
f.f.l.Concatenate lists
Victim
f.f.l.
mod.l.
Buffered Frames
* Remove BufferedFrames from both Lists
(make them not buffered)
-
8/6/2019 OS Project 2011 Documentation
9/29
9
Details
env_create( )
in kern/user_environment.c (no student code needed, just understand)
After env_create() is executed to load a program binary, the binary will partially loaded in main memory (part of
segments + stack page) and FULLY loaded in page file (all segments + stack page)
Figure 3: View of working set after loading a program binary with env_create(). [(1): * this asterisk means that
Used Bit is set to 1 in page table for this virtual address, (2) WS_Last_index: the WS index to start iterating
from (used in clock algorithm)]
Fault Handler
In function fault_handler() , " kern/trap.c"
Fault: is an exception thrown by the processor (MMU) to indicate that:o A pagecant be accessed due to either its not present in the main memory ORo A page table is not present in the main memory. (see the following figure)
BIN
Segment 1, page 1
Segment 1, page 2
Segment 1, page 3
Segment 2, page 1
Segment 2, page 2
201000 *(1)
202000 *
203000 *
801000 *
Program Binary
Segments and pages
Working Set after
loading
(WS_MAX_SIZE = 5)
Loaded in virtual address 201000
Loaded in virtual address 202000
Loaded in virtual address 203000
NOT Loaded in
main memory
USTAKTOPPAGESIZE *
First Stack Page
Loaded in virtual address 801000
WS_Last_index (2)
-
8/6/2019 OS Project 2011 Documentation
10/29
1
Figure 4: Fault types (table fault and page fault)
You should handle any fault in FOS kernel by:a. Allocate a new page (or a new page table) for this faulted page in the main memoryb. Loading the faulted page (or its page table) back to main memory from page file (secondary
memory) if the page (or the table) exists in page file. Otherwise, add it to the page file. You can handle the page (or page table) fault in function fault_handler() in trap.c, as follows
LinearAddress
Page Directory
Dir Entry
Page Table
Table Entry
CR3
32 bits
P=1
LinearAddress
Page Directory
Dir Entry
CR332 bits
4 KB
P=0
TABLEFault
10 bits10 bits 12 bits
DIR PAGE OFFSET
10 bits10 bits 12 bits
DIR PAGE OFFSET
P=0
PAGEFault
CASE1: Table not exist CASE2: Page not exist
If the directory entry of the faulted address is NOT PRESENT then
Handle Table Fault
Else
Handle Page Fault
-
8/6/2019 OS Project 2011 Documentation
11/29
Table fault handler should be written as follows: if the size of the table working set < TABLE_WS_MAX_SIZEthen do
Placement:
1.save the directory entry of the faulted address to use it in step 32.allocate a new frame for the table and link it with the directory entry of the faulted address(create_page_table() function will do this for you)3.if saved directory entry = NULL then
1. This is a new table (i.e. not exist in page file)2. So, do nothing here.
4.else1. The table of the faulted address exists in the page file2. So, read it from page file to memory3. Remove it from page file as we will never read it again from page file.
5.update the table working set else, do
Replacement:
1.implement clock algorithm to find the virtual address of the victim table from table WS toreplace (refer toAPPENDIX II)
2.write the victim table in the page file3.remove the victim table by
1. unlink it from the directory entry by setting PRESENT BIT to 0 ( IMPORTANT NOTE:Dont set the whole directory entry to 0 (NULL), this means that the victim table
doesnt exist in both memory and page file, which is not the case here)
2. freeing its frame3. and remove it from the table working set
4.Apply Placement steps here
-
8/6/2019 OS Project 2011 Documentation
12/29
2
Page fault handler should work as follows:
Refer to helper functions to deal with flags of Page Table (or page Directory) entries (Appendix III) Refer to the linked lists, basic and helper memory manager functions (Appendix V)
o if the size of the page working set < PAGE_WS_MAX_SIZEthen doPlacement:
1. check if required page is buffered (seeAPPENDIX III), if true, do the following:o
Just reclaim the page by setting present bit to 1, and buffered bit to 0, and set itsframe_info->isBufferedto 0. (seeAPPENDIX III, and IV)
o remove its Frame_Info from: "modified_frame_list" if the page was modified(modified bit = 1), or from "free_frame_list" if the page was not modified (modified
bit = 0) (seeAPPENDIX III and IV)
elseo allocate and map a frame for the faulted pageo read the faulted page from page file to memory
If the page does not exist on page file, then CHECK if it is a stack page. If it is a
new stack page, add a new empty page with this faulted address to page file
(refer toAppendix I and II)
3. update the page working set else, do
Replacement:
4. implement clock algorithm to find the victim virtual address from page working set5. for the victim page:
Prepare its Frame_Info by flagging it as buffered, setting the environment and thevictim virtual address inside this Frame_Info Set the buffered bit to 1 in the victim page table Set the present bit to 0 in the victim page table
6. If the victim page was not modified, then: Add the victim frame to "free_frame_list" at the tail (seeAPPENDIX IV)
Else Add the victim frame to "modified_frame_list" at the tail (seeAPPENDIX IV)
Check the count of modified pages in "modified_frame_list", if it reachesMAX_MODIFIED_LIST_COUNT, reset the counter and do the following:
1. for each frame in the "modified_frame_list"a. update its page in page file (seeAPPENDIX I),b. set the modified bit of its page in its page table to 0
2. Concatenate the "modified_frame_list" to "free_frame_list" (by doing this,the modified list will be cleared) (see helper functions inAPPENDIX V)
2.Apply Placement steps described before
-
8/6/2019 OS Project 2011 Documentation
13/29
3
env_free( )
in kern/user_environment.c
For each BUFFERED page in the User Spaceo If the buffered page is MODIFIED then
Remove its "Frame_Info" from the "modified_frame_list" Flag its "Frame_Info" as NOT Buffered (setting isBuffered to 0) Add it to the "free_frame_list"
o Else Just flag its "Frame_Info" as NOT Buffered (setting isBuffered to 0)
Then, you should free :o The pages in the page working seto The page tables the table working seto The directory table
Frees all pages and tables from page file, this code is alreadywritten for youDynamic allocation/free
Functions malloc() and freeHeap() in "lib/malloc.c"
You will need to implement a dynamic allocation function for your user programs to make runtimeallocations, also a free Heap function to free ALL the heap
Your should implement function malloc() (allocates user memory) and freeHeap()(frees all userheap) functions in lib/malloc.c
The user should call the kernel to allocate/free memory for it. But wait a minute!! remember that the user code is not the kernel (i.e. when a user code is
executing, the processor runs user mode (less privileged mode), and to execute kernel code the
processor need to go from user mode to kernel mode)
The switch from user mode to kernel mode is done by a software interruptcalled System Call. All you need to do is to call a function prefixed sys_ from your user code to call the kernel code
that does the job, (e.g. from user function malloc(), call sys_allocateMem() which then will call
kernel function allocateMem() to allocate memory for the user) as shown in figure:
-
8/6/2019 OS Project 2011 Documentation
14/29
4
Figure 5: Sequence diagram of dynamic allocation using malloc()
Your kernel code will be in allocateMem() and freeMem() functions in memory_manager.c In allocateMem(), all pages you allocate will not be added to the working set (not exist in main
memory), but they should added to the page file, so that when this page is accessed, a page fault
will load it to memory.
In freeMem(), you will need to:o For each BUFFERED page in the Given Range
If the buffered page is MODIFIED then Remove its "Frame_Info" from the "modified_frame_list" Flag its "Frame_Info" as NOT Buffered (setting isBuffered to 0) Add it to the "free_frame_list"
Else Just flag its "Frame_Info" as NOT Buffered (setting isBuffered to 0)
Set its entry in the page table to NULLo
Remove ONLY working set pages and working set page tables that are located in the givenuser virtual address range. (REMEMBER to update the working sets after removing)
o Remove all pages and all tables in the given range from the page file.
Switch to
Kernel mode
Call
Switch back to
User mode
Call *USERMalloc
KERNELallocateMem *
S/W INTsys_allocateMem()
NOTE: You should do the (*) operations only
-
8/6/2019 OS Project 2011 Documentation
15/29
5
BONUSES
First: Freeing RAM when it's Full
When allocating new frame, if there's no free frame, then you should:
1- Free some/all frames from the modified page list and add them to the free frame list2- If the modified list is empty, then you should free at least 1 frame from the user working set by applying the
clock algorithm
Second: Buffering Page Tables
Sometimes during the execution, the program needs a large number of page tables to be loaded in memory at the
same time to use them all. This can lead to large number of TABLE FAULTS.
Since the page tables are always modified (due to continuous changes in status bits), any fault on it cause it to be
written back in the page file (H.D.D) which consume a lot of time and cause the CPU to be idle.
In order to solve this problem, we can apply the BUFFERING concept on the page tables just as we apply it for pages.Your task is to define the required variables in your code and maintain them during the whole code in order to
manage the PAGE TABLE BUFFERING.
CHALLENGE!!In fault_handler() function you are required to write TABLE replacement algorithm using NORMAL CLOCK, normal
clock works with 1 used bitof the table, however, it does not achieve good level of fairness when choosing a victim!.
and it will generate a lot of table faults due to
1- It doesnt care about the pages that each page table refer to2- It deal with all tables as a same regardless how many times each table is used
These two issues may lead to remove table that is either frequently used or contains pages that are currently
used.
Can you write a better algorithm?
Challenge Description
Develop a table replacement algorithm that minimizes the number of table faults occur when testing FOS project.
Every effort is welcomed, search the books, ask faculty staff, search the
internet or even invent your own algorithm.
-
8/6/2019 OS Project 2011 Documentation
16/29
6
Testing
How to test your project
To test your implementation, you need to run each of the following test programs. Each program should
test certain part from the project. In addition, after completing all parts, you can test the whole project
using quickSort program. User programs found in "user/" folder:
FIRST: Testing Each Part
If a test succeeds, it will print and success message on the screen, otherwise the test will panic at the error
line and display it on the screen
Testing Page & Table Placement:
test_ptf_stack.c: tests page faults on stack + page & table placement. You need to initialize the Working
set sizes as follows:
PAGE_WS_MAX_SIZE = 20
TABLE_WS_MAX_SIZE = 20
Testing Page Replacement:
test_pf_clock.c : to test your implementation for the clock algorithm for page replacement. You need to
initialize the Working set sizes as follows:
PAGE_WS_MAX_SIZE = 8
TABLE_WS_MAX_SIZE = 20
test_pf_stack.c: to test your implementation for handling the new stack pages and the modified pages.
You need to initialize the Working set sizes as follows:
PAGE_WS_MAX_SIZE = 8
TABLE_WS_MAX_SIZE = 20
Testing Table Replacement:
test_table_replacement.c, test_table_replacement_2.c:tests page faults on stack + table replacement
(clock algorithm and dealing with page file for writing and reading tables)
PAGE_WS_MAX_SIZE = 30
TABLE_WS_MAX_SIZE = 3
Testing Page & Table Replacement:
test_ptf_stack.c : tests page faults on stack + page/table replacement.
PAGE_WS_MAX_SIZE = 4
TABLE_WS_MAX_SIZE = 3
-
8/6/2019 OS Project 2011 Documentation
17/29
7
Testing Dynamic Allocation & Free:
test_malloc.c : to test your implementation for the malloc() and allocateMem() functions. You need to
initialize the Working set sizes as follows:
PAGE_WS_MAX_SIZE = 15
TABLE_WS_MAX_SIZE = 15
test_freeHeap.c : to test your implementation for the freeHeap() and freeMem() functions. You need to
initialize the Working set sizes as follows:
PAGE_WS_MAX_SIZE = 15
TABLE_WS_MAX_SIZE = 5
SECOND: Testing Whole Project
You should run each of the following scenarios successfully
Scenario 1: Env_Free1. Make sure that PAGE_WS_MAX_SIZE = 10, and TABLE_WS_MAX_SIZE = 202. run command meminfo
denote free frames as FF1
3. Run user program fos_add4. kill fos_add5. run command meminfo
make sure that free frames = FF1
Scenario 2: Small Programs
All other programs in the "user/" folder should be run successfully with the following Working set sizes
PAGE_WS_MAX_SIZE = 4
TABLE_WS_MAX_SIZE = 3
Scenario 3: QuickSort Program
1. Make sure that PAGE_WS_MAX_SIZE = 4, TABLE_WS_MAX_SIZE = 32. Run user program quicksort_heap and test it according to the following steps:
Number of Elements = 3,000
Initialization method : Ascending
Do you want to repeat (y/n) : y
3. The program should sort the array successfully
-
8/6/2019 OS Project 2011 Documentation
18/29
8
Scenario 4: QuickSort_freeHeap Program
1. Make sure that PAGE_WS_MAX_SIZE = 500, and TABLE_WS_MAX_SIZE = 202. run user program fos_helloWorld3. run user program fos_add4. run user program quicksort_heap_freeHeap and test it according to the following steps:
Number of Elements = 10,000
Initialization method : Semi randomDo you want to repeat (y/n) : y
Number of Elements = 100,000
Initialization method : Semi random
Do you want to repeat (y/n) : y
Number of Elements = 1,000
Initialization method : Ascending
Do you want to repeat (y/n) : y
Number of Elements = 300,000
Initialization method : Semi randomDo you want to repeat (y/n) : n
At each step, the program should sort the array successfully and DONT panic
Enjoy your OS
GOOD LUCK
-
8/6/2019 OS Project 2011 Documentation
19/29
9
APPENDICES
APPENDIX I: Page File Helper FunctionsThere are some functions that help you work with the page file. They are declared and defined in
kern/file_manager.h and kern/file_manager.c respectively. Following is brief description about those functions:
Add a new environment page to the page file
Description:
Add a new environment page with the given virtual address to the page file and initialize it by zeros.
Function declaration:
int pf_add_empty_env_page( struct Env* ptr_env, uint32 virtual_address);
Parameters:
ptr_env: pointer to the environment that you want to add the page for it.
virtual_address: the virtual address of the page to be added.
Return value:
= 0: the page is added successfully to the page file.
= E_NO_PAGE_FILE_SPACE: the page file is full, cant add any more pages to it.
Example:
In dynamic allocation: let for example we want to dynamically allocate 1 page at the beginning of the
heap (i.e. at address USER_HEAP_START), so we need to add this page to the page file as follows:
int ret = pf_add_empty_env_page(ptr_env, USER_HEAP_START);if (ret == E_NO_PAGE_FILE_SPACE)
panic(ERROR: No enough virtual space on the page file);
Read an environment page from the page file to the main memory
Description:
Read an existing environment page at the given virtual address from the page file.
Function declaration:
int pf_read_env_page(struct Env* ptr_env, void *virtual_address);
Parameters:
ptr_env: pointer to the environment that you want to read its page from the page file.
virtual_address: the virtual address of the page to be read.
Return value:
= 0: the page is read successfully to the given virtual address of the given environment.
= E_PAGE_NOT_EXIST_IN_PF: the page doesnt exist on the page file (i.e. no one added it before to
the page file).
Example:
In placement steps: let for example there is a page fault occur at certain virtual address, then, we
want to read it from the page file and place it in the main memory at the faulted virtual address as
follows:
-
8/6/2019 OS Project 2011 Documentation
20/29
21
int ret = pf_read_env_page(ptr_env, fault_va);
if (ret == E_PAGE_NOT_EXIST_IN_PF){
...}
Update certain environment page in the page file by contents from the main memory
Description:
Updates an existing page in the page file by the given frame in memory
Function declaration:
int pf_update_env_page(struct Env* ptr_env, void *virtual_address, structFrame_Info* modified_page_frame_info));
Parameters:
ptr_env: pointer to the environment that you want to update its page on the page file.
virtual_address: the virtual address of the page to be updated.
modified_page_frame_info: the Frame_Info* related to this page.
Return value:
= 0: the page is updated successfully on the page file.
= E_PAGE_NOT_EXIST_IN_PF: the page to be updated doesnt exist on the page file (i.e. no one add
it before to the page file).
Example:
structFrame_Info *ptr_frame_info = get_frame_info();
int ret = pf_update_env_page(environment, virtual_address, ptr_frame_info);
Read an environment table from the page file to the main memory
Description:
Read an existing environment table at the given virtual address from the page file memory.
Function declaration:
intpf_read_env_table(struct Env* ptr_env, uint32 virtual_address, uint32*
tableKVirtualAddress);
Parameters:
ptr_env: pointer to the environment that you want to read its page from the page file.
virtual_address: the first user virtual address of the table (in userspace)
tableKVirtualAddress: the kernel virtual address of the table needed to be read from page file
Return value:
= 0: the table is read successfully to the given table virtual address.
= E_TABLE_NOT_EXIST_IN_PF: the table doesnt exist on the page file
Example:
Assume for example that there is a table fault at certain virtual address, then, we want to read table
from the page file and place it in the working set in main memory:
uint32* ptr_table;create_page_table(faulted_Env->env_pgdir, fault_va, &ptr_table);
pf_read_env_table(ptr_env, fault_va, ptr_table);
-
8/6/2019 OS Project 2011 Documentation
21/29
2
Write a certain environment table to the page file
Description:
Write an environment page table to the page file.
Function declaration:
intpf_write_env_table(struct Env* ptr_env, void *virtual_address, uint32*tableKVirtualAddress);
Parameters:
ptr_env: pointer to the environment that you want to write its page to the page file.
virtual_address: the first user virtual address of the table (in userspace)
tableKVirtualAddress: the kernel virtual address of the table needed to be written to page file
Return value:
= 0: the table is written successfully to the page file
= E_NO_PAGE_FILE_SPACE: the page file is full, cant add any more pages (tables) to it.
Example:
//write a table to page fileuint32* ptr_table;get_page_table(faulted_env->env_pgdir, (void*) virtual_address,&ptr_table(;int success = pf_write_env_table(curenv, virtual_address, ptr_table);
Remove an existing environment page/table from the page file
Description:
Remove an existing environment page/table at the given virtual address from the page file.
Function declaration:
void pf_remove_env_page(struct Env* ptr_env, uint32 *virtual_address);
void pf_remove_env_table(struct Env* ptr_env, uint32 virtual_address);
Parameters:
ptr_env: pointer to the environment that you want to remove its page (or table) on the page file.
virtual_address (page): the virtual address of the page to be removed.
virtual_address (table): the first user virtual address of the table (in userspace)
Example:
Lets assume for example we want to free 1 page at the beginning of the heap (i.e. at address
USER_HEAP_START), so we need to remove this page from the page file as follows:
pf_remove_env_page(ptr_env, USER_HEAP_START);
Or that we want to free 1 table at the beginning of the user heap (i.e. at address
USER_HEAP_START), so we need to remove this table from the page file as follows:
pf_remove_env_table(ptr_env, USER_HEAP_START);
-
8/6/2019 OS Project 2011 Documentation
22/29
22
APPENDIX II: Working Set Structure & Helper Functions
Working Set Structure
As stated before, each environment has two working sets with FIXED size that hold the virtual addresses
corresponding to the pages/tables that are loaded in main memory.
The working set is defined as a fixed size array of virtual addresses inside the environment structure structEnvlocated in inc/environment_definitions.h.
Figure 6: Definitions of 2 working sets & their indices inside struct Env
Working Set Functions
These functions are declared and defined in kern/memory_manager.h and kern/ memory_manager.c
respectively. Following are brief description about those functions:
inline uint32 env_page_ws_get_size(structEnv *e)inline uint32 env_table_ws_get_size(structEnv *e)
Description:
Counts the pages/tables loaded in main memory of a given environment
Parameters:
e: pointer to the environment that you want to count its working set size
Return value:
Number of pages/tables loaded in main memory for environment e,(i.e. e working set size)
#define PAGE_WS_MAX_SIZE 5#define TABLE_WS_MAX_SIZE 5
struct WorkingSetElement {uint32 virtual_address; // the virtual address of the pageuint8 empty; // if empty = 0, the entry is valid, if empty = 1 , entry is
empty or free};
struct Env {...//page working set management, array of PAGE_WS_MAX_SIZE entriesstruct WorkingSetElement pageworkingSet[PAGE_WS_MAX_SIZE];
//table working set management, array of TABLE_WS_MAX_SIZE entriesstruct WorkingSetElement tableworkingSet[TABLE_WS_MAX_SIZE];
// used for modified clock algorithm, the next item (page) pointeruint32 page_WS_last_index;
// used for clock algorithm, the next item (table) pointeruint32 table_WS_last_index;};
-
8/6/2019 OS Project 2011 Documentation
23/29
23
inlinevoid env_page_ws_invalidate(structEnv* e, uint32 virtual_address)
inlinevoid env_table_ws_invalidate(structEnv* e, uint32 virtual_address)
Description:
Removes page/table entry at the given virtual address from the working set ofe
Parameters:
e: pointer to an environmentvirtual_address: the virtual address to remove from working set
inlinevoid env_page_ws_set_entry(structEnv* e, uint32 entry_index, uint32 virtual_address)
inlinevoid env_table_ws_set_entry(structEnv* e, uint32 entry_index, uint32 virtual_address)
Description:
Sets the entry number entry_index in eworking set to given virtual address after ROUNDING it
DOWN to the start of page (or page table)
Parameters:
e: pointer to an environment
entry_index: the working set entry index to set the given virtual address
virtual_address: the virtual address to set
inlinevoid env_page_ws_clear_entry(structEnv* e, uint32 entry_index)
inlinevoid env_table_ws_clear_entry(structEnv* e, uint32 entry_index)
Description:
Clears (make empty) the entry at entry_index in eworking set.
Parameters:
e: pointer to an environment
entry_index: working set entry index
inline uint32 env_page_ws_get_virtual_address(structEnv* e, uint32 entry_index)
inline uint32 env_table_ws_get_virtual_address(structEnv* e, uint32 entry_index)
Description:
Returns the virtual address of the page at entry entry_index in environment eworking set
Parameters:
e: pointer to an environment
entry_index: working set entry index
Return value:
The virtual address ofthe page at entry entry_index in environment eworking set
-
8/6/2019 OS Project 2011 Documentation
24/29
24
inline uint32 env_page_ws_is_entry_empty(structEnv* e, uint32 entry_index)
inline uint32 env_table_ws_is_entry_empty(structEnv* e, uint32 entry_index)
Description:
Returns a value indicating whether the entry at entry_index in environment e working set is
empty
Parameters:
e: pointer to an environment
entry_index: working set entry index
Return value:
0: if the working set entry at entry _index is NOT empty
1: if the working set entry at entry _index is empty
inlinevoidenv_page_ws_print(structEnv* e)
inline voidenv_table_ws_print(structEnv* e)
Description:
Print the page (or table) working set together with the used and modified bits. It also show where
the last_WS_indexof the working set is point to.
Parameters:
e: pointer to an environment
-
8/6/2019 OS Project 2011 Documentation
25/29
25
APPENDIX III: Functions for manipulating permissions in page tables and
directory
inline void pt_set_page_permissions(struct Env* ptr_env, uint32 virtual_address, uint32
permissions_to_set, uint32 permissions_to_clear)
Description:
Sets the permissions given by permissions_to_setto 1 in the page table entry of the given
page (virtual address), and clears the permissions given by permissions_to_clear. The
environment used is the one given by ptr_pgdir
Parameters:
ptr_env: pointer to environment that you should work on
virtual_address: any virtual address of the page
permissions_to_set : page permissions to be set to 1
permissions_to_clear : page permissions to be set to 0
Return value: none
Example1: to set page PERM_BUFFERED bit to 1 and set PERM_PRESENT to 0
Example2: to set PERM_MODIFIED to 0
inline uint32 pt_get_page_permissions(struct Env* ptr_env, uint32 virtual_address )
Description:
Returns all permissions bits for the given page (virtual address) in the given environment page
directory (ptr_pgdir)
Parameters:
ptr_env: pointer to environment that you should work on
virtual_address: any virtual address of the page
Return value:
Unsigned integer containing all permissions bits for the given page
Example: To check if a page is modified:
uint32 page_permissions = pt_get_page_permissions(environment,virtual_address);if(page_permissions & PERM_MODIFIED){
. . .}
pt_set_page_permissions(environment, virtual_address, 0,PERM_MODIFIED);
pt_set_page_permissions(environment, virtual_address,PERM_BUFFERED, PERM_PRESENT);
-
8/6/2019 OS Project 2011 Documentation
26/29
26
inline uint32 pd_is_table_used(uint32* ptr_pgdir, uint32 virtual_address)
Description:
Returns a value indicating whether the table at virtual_address in was used by the processor
Parameters:
ptr_pgdir: pointer to environment page directory
virtual_address: any virtual address inside the table
Return value:
0: if the table at virtual_address is not used (accessed) by the processor
1: if the table at virtual_address is used (accessed) by the processor
Example:
inlinevoid pd_set_table_unused(uint32* ptr_pgdir, uint32 virtual_address)
Description:
Clears the Used Bit of the table at virtual_address
Parameters:
ptr_pgdir: pointer to environment page directory
virtual_address: any virtual address inside the table
if(pd_is_table_used(faulted_env->pgdir, virtual_address)){
}
-
8/6/2019 OS Project 2011 Documentation
27/29
27
APPENDIX IV: Page Buffering
Structures
In order to keep track of the buffered frames, the following changes were added to FOS:
1- We use one of the available bits in the page table as a BUFFERED bit (Bit number 9) to indicate whether
the frame of this page is buffered or not.
2- Add the following information to the "Frame_Info" structure:
a- isBuffered: to indicate whether this frame is buffered or not
b-va: virtual address of the page that was mapped to this buffered frame
c- environment: the environment that own this virtual address
3- Create a new list called "modified_frame_list"
A new Frame_Info* linked list is added to FOS to keep track of buffered modified page frames,(see
memory_manager.c):
Functions
inline void bufferList_add_page(struct Linked_List* bufferList,struct Frame_Info
*ptr_frame_info)
inline void bufferlist_remove_page(struct Linked_List* bufferList, struct Frame_Info
*ptr_frame_info)
Description:
Adds/Removes the given frame (ptr_frame_info) to/from the given list (bufferList)
Parameters:
bufferList: pointer to a linked list of Frame_Info* elements
ptr_frame_info: a pointer to Frame_Info object that will be added/removedExample:
struct Linked List modified frame list
0111231 123456789
PAGE FRAME ADDRESS 12...31 PA 0 0U/S
R/W
DAVAIL 0 0
P: Present bit (PERM_PRESENT)R/W: Read/Write bit (PERM_WRITABLE)U/S: User/Supervisor bit (PERM_USER)A: Accessed bit (PERM_USED)D: Dirty bit (PERM_MODIFIED)B: Buffered bit (PERM_BUFFERED)AVAIL: Available for system programmers useNOTE: 0 indicates Intel reserved. Dont define.
B
struct Frame_Info* ptr_victim_frame;. . .
bufferList_add_page(&modified_page_list, ptr_victim_frame);
-
8/6/2019 OS Project 2011 Documentation
28/29
28
Appendix V: Basic and Helper Memory Management Functions
Basic Functions
The basic memory manager functions that you may need to use are defined in kern/memory_manager.c file:
Function Name Description
allocate_frame Used to allocate a free frame from the free frame list
free_frame Used to free a frame by adding it to free frame list
map_frameUsed to map a single page with a given virtual address into a given allocated frame,
simply by setting the directory and page table entries
get_page_tableUsed by map_frame to get a pointer to the page table if exist, or creating it if
not exist
unmap_frameUsed to un-map a frame at the given virtual address, simply by clearing the page
table entry
get_frame_infoUsed by unmap_frame to get both the page table and the frame of the given
virtual address
Helpers Functions
There are some helper functions that we may need to use them in the rest of the course:
Function Description Defined in
LIST_FOREACH
(Frame_Info*,
Linked_List *)
Used to traverse a linked list.
Example: to traverse the modified list
struct Frame_Info* ptr_frame_info = NULL;
LIST_FOREACH(ptr_frame_info, &modified_frame_list)
{
.
}
inc/queue.h
LIST_CONCAT
(Linked_List* list1,Linked_List* list2)
Used to concatenate list2 to list1, after concatenation
list2 will be empty, and list 1 = list1+list2 inc/queue.h
to_frame_number
(Frame_Info *)
Return the frame number of the corresponding
Frame_Info elementKern/memory_manager.h
to_physical_address
(Frame_Info *)
Return the start physical address of the frame
corresponding to Frame_Info elementKern/memory_manager.h
to_frame_info
(uint32 phys_addr)
Return a pointer to the Frame_Info corresponding to
the given physical addressKern/memory_manager.h
K_PHYSICAL_ADDRESS(uint32 kernel
virtual address)
Used for returning the physical address of a givenkernel virtual address after KERNEL_BASE. This is done
simply by adding ( KERNEL_BASE) to the given address.
Kern/helpers.h
-
8/6/2019 OS Project 2011 Documentation
29/29
Function Description Defined in
K_VIRTUAL_ADDRESS
(uint32 physical
address)
Used for returning the kernel virtual address of a given
physical address. This is done simply by adding
KERNEL_BASE to the given address.
Kern/helpers.h
PDX (uint32 virtual
address)
Gets the page directory index in the given virtual
address (10 bits from 22 31).Inc/mmu.h
PTX (uint32 virtual
address)
Gets the page table index in the given virtual address
(10 bits from 12 21).Inc/mmu.h
ROUNDUP
(uint32 value, uint32
align)
Rounds a given value to the nearest upper value that
is divisible by align. Inc/types.h
ROUNDDOWN
(uint32 value, uint32
align)
Rounds a given value to the nearest lower value that
is divisible by align. Inc/types.h
tlb_invalidate(uint32*
page_directory,
uint32 virtual
address)
Refresh the cache memory (TLB) to remove the givenvirtual address from it.
Kern/helpers.c
rcr3()Read the physical address of the current page directory
which is loaded in CR3Inc/x86.h
lcr3(uint32 physical
address of directory)
Load the given physical address of the page directory
into CR3Inc/x86.h
Have a nice project
GOOD LUCK