Accessibility for data analysis: using digital tools to unlock archives … · 2020. 1. 17. ·...
Transcript of Accessibility for data analysis: using digital tools to unlock archives … · 2020. 1. 17. ·...
![Page 1: Accessibility for data analysis: using digital tools to unlock archives … · 2020. 1. 17. · using digital tools to unlock archives potential Kathryn Dan, ANU and Ingrid Mason,](https://reader035.fdocuments.us/reader035/viewer/2022071011/5fc8dd161bb26b70ea04bd78/html5/thumbnails/1.jpg)
Accessibility for data analysis:
using digital tools to unlock
archives potential
Kathryn Dan, ANU and Ingrid Mason, AARNET
![Page 2: Accessibility for data analysis: using digital tools to unlock archives … · 2020. 1. 17. · using digital tools to unlock archives potential Kathryn Dan, ANU and Ingrid Mason,](https://reader035.fdocuments.us/reader035/viewer/2022071011/5fc8dd161bb26b70ea04bd78/html5/thumbnails/2.jpg)
2
ANU Archives Program
![Page 3: Accessibility for data analysis: using digital tools to unlock archives … · 2020. 1. 17. · using digital tools to unlock archives potential Kathryn Dan, ANU and Ingrid Mason,](https://reader035.fdocuments.us/reader035/viewer/2022071011/5fc8dd161bb26b70ea04bd78/html5/thumbnails/3.jpg)
Outline
• Previous digitisation projects
• The archives: Sydney Stock Exchange
Registers
• Project objectives and partners
• Work so far … and where to next
3
![Page 4: Accessibility for data analysis: using digital tools to unlock archives … · 2020. 1. 17. · using digital tools to unlock archives potential Kathryn Dan, ANU and Ingrid Mason,](https://reader035.fdocuments.us/reader035/viewer/2022071011/5fc8dd161bb26b70ea04bd78/html5/thumbnails/4.jpg)
4
![Page 5: Accessibility for data analysis: using digital tools to unlock archives … · 2020. 1. 17. · using digital tools to unlock archives potential Kathryn Dan, ANU and Ingrid Mason,](https://reader035.fdocuments.us/reader035/viewer/2022071011/5fc8dd161bb26b70ea04bd78/html5/thumbnails/5.jpg)
5
![Page 6: Accessibility for data analysis: using digital tools to unlock archives … · 2020. 1. 17. · using digital tools to unlock archives potential Kathryn Dan, ANU and Ingrid Mason,](https://reader035.fdocuments.us/reader035/viewer/2022071011/5fc8dd161bb26b70ea04bd78/html5/thumbnails/6.jpg)
6
![Page 7: Accessibility for data analysis: using digital tools to unlock archives … · 2020. 1. 17. · using digital tools to unlock archives potential Kathryn Dan, ANU and Ingrid Mason,](https://reader035.fdocuments.us/reader035/viewer/2022071011/5fc8dd161bb26b70ea04bd78/html5/thumbnails/7.jpg)
Project objectives• Demonstrate data extraction to enhance
research potential of digitised archival resources
• Test digitisation workflows and data transfer
• Test applicability of automated handwriting recognition
• Investigate automating description
• Experiment with providing archival collections as ‘data’
• Provide a case study for future work
7
![Page 8: Accessibility for data analysis: using digital tools to unlock archives … · 2020. 1. 17. · using digital tools to unlock archives potential Kathryn Dan, ANU and Ingrid Mason,](https://reader035.fdocuments.us/reader035/viewer/2022071011/5fc8dd161bb26b70ea04bd78/html5/thumbnails/8.jpg)
8
![Page 9: Accessibility for data analysis: using digital tools to unlock archives … · 2020. 1. 17. · using digital tools to unlock archives potential Kathryn Dan, ANU and Ingrid Mason,](https://reader035.fdocuments.us/reader035/viewer/2022071011/5fc8dd161bb26b70ea04bd78/html5/thumbnails/9.jpg)
9
Search/Browse metadata
![Page 10: Accessibility for data analysis: using digital tools to unlock archives … · 2020. 1. 17. · using digital tools to unlock archives potential Kathryn Dan, ANU and Ingrid Mason,](https://reader035.fdocuments.us/reader035/viewer/2022071011/5fc8dd161bb26b70ea04bd78/html5/thumbnails/10.jpg)
10
![Page 11: Accessibility for data analysis: using digital tools to unlock archives … · 2020. 1. 17. · using digital tools to unlock archives potential Kathryn Dan, ANU and Ingrid Mason,](https://reader035.fdocuments.us/reader035/viewer/2022071011/5fc8dd161bb26b70ea04bd78/html5/thumbnails/11.jpg)
11
![Page 12: Accessibility for data analysis: using digital tools to unlock archives … · 2020. 1. 17. · using digital tools to unlock archives potential Kathryn Dan, ANU and Ingrid Mason,](https://reader035.fdocuments.us/reader035/viewer/2022071011/5fc8dd161bb26b70ea04bd78/html5/thumbnails/12.jpg)
12
![Page 13: Accessibility for data analysis: using digital tools to unlock archives … · 2020. 1. 17. · using digital tools to unlock archives potential Kathryn Dan, ANU and Ingrid Mason,](https://reader035.fdocuments.us/reader035/viewer/2022071011/5fc8dd161bb26b70ea04bd78/html5/thumbnails/13.jpg)
Steps1. Digitise and extract
2. Convert ledger to data – columns, rows
3. Recognise text – transcription and automation
4. Automate recognition
5. Develop and test further extraction https://nbviewer.jupyter.org/github/wragge/sydney-stock-
exchange/blob/master/summary.ipynb
13
![Page 14: Accessibility for data analysis: using digital tools to unlock archives … · 2020. 1. 17. · using digital tools to unlock archives potential Kathryn Dan, ANU and Ingrid Mason,](https://reader035.fdocuments.us/reader035/viewer/2022071011/5fc8dd161bb26b70ea04bd78/html5/thumbnails/14.jpg)
14
Extraction
![Page 15: Accessibility for data analysis: using digital tools to unlock archives … · 2020. 1. 17. · using digital tools to unlock archives potential Kathryn Dan, ANU and Ingrid Mason,](https://reader035.fdocuments.us/reader035/viewer/2022071011/5fc8dd161bb26b70ea04bd78/html5/thumbnails/15.jpg)
15
![Page 16: Accessibility for data analysis: using digital tools to unlock archives … · 2020. 1. 17. · using digital tools to unlock archives potential Kathryn Dan, ANU and Ingrid Mason,](https://reader035.fdocuments.us/reader035/viewer/2022071011/5fc8dd161bb26b70ea04bd78/html5/thumbnails/16.jpg)
16
Find straight lines in the edges
# Note that the minLineLength and
maxLineGap values can have a dramatic effect
on the number of lines detected.
# Note too that what looks to the human
eye as a single straight line,
# can actually be a series of short line
with tiny gaps between them,
# so while increasing the minLineLength
reduces noise from text, it can also filter
out columns.
![Page 17: Accessibility for data analysis: using digital tools to unlock archives … · 2020. 1. 17. · using digital tools to unlock archives potential Kathryn Dan, ANU and Ingrid Mason,](https://reader035.fdocuments.us/reader035/viewer/2022071011/5fc8dd161bb26b70ea04bd78/html5/thumbnails/17.jpg)
• directory
• name
• path
• referenceCode
• startDate
• endDate
• year
• width – width of the image
• height – height of the image
• columns – number of columns
• column_positions – comma-separated string of x values of columns in pixels
17
![Page 18: Accessibility for data analysis: using digital tools to unlock archives … · 2020. 1. 17. · using digital tools to unlock archives potential Kathryn Dan, ANU and Ingrid Mason,](https://reader035.fdocuments.us/reader035/viewer/2022071011/5fc8dd161bb26b70ea04bd78/html5/thumbnails/18.jpg)
3 41076
4 26917
2 4825
1 19
0 6
Name: columns, dtype: int64
18
![Page 19: Accessibility for data analysis: using digital tools to unlock archives … · 2020. 1. 17. · using digital tools to unlock archives potential Kathryn Dan, ANU and Ingrid Mason,](https://reader035.fdocuments.us/reader035/viewer/2022071011/5fc8dd161bb26b70ea04bd78/html5/thumbnails/19.jpg)
19
![Page 20: Accessibility for data analysis: using digital tools to unlock archives … · 2020. 1. 17. · using digital tools to unlock archives potential Kathryn Dan, ANU and Ingrid Mason,](https://reader035.fdocuments.us/reader035/viewer/2022071011/5fc8dd161bb26b70ea04bd78/html5/thumbnails/20.jpg)
20
![Page 21: Accessibility for data analysis: using digital tools to unlock archives … · 2020. 1. 17. · using digital tools to unlock archives potential Kathryn Dan, ANU and Ingrid Mason,](https://reader035.fdocuments.us/reader035/viewer/2022071011/5fc8dd161bb26b70ea04bd78/html5/thumbnails/21.jpg)
21
![Page 22: Accessibility for data analysis: using digital tools to unlock archives … · 2020. 1. 17. · using digital tools to unlock archives potential Kathryn Dan, ANU and Ingrid Mason,](https://reader035.fdocuments.us/reader035/viewer/2022071011/5fc8dd161bb26b70ea04bd78/html5/thumbnails/22.jpg)
22
![Page 23: Accessibility for data analysis: using digital tools to unlock archives … · 2020. 1. 17. · using digital tools to unlock archives potential Kathryn Dan, ANU and Ingrid Mason,](https://reader035.fdocuments.us/reader035/viewer/2022071011/5fc8dd161bb26b70ea04bd78/html5/thumbnails/23.jpg)
23
[{'task': 'T4',
'task_label': 'Can you see a handwritten date?',
'value': 'Yes'},
{'task': 'T0',
'task_label': 'Can you identify the session? Look for one of the following handwritten headings.',
'value': 'No session heading'},
{'task': 'T5',
'task_label': 'What is the day of the week?',
'value': 'Saturday'},
{'task': 'T1',
'value': '29',
'task_label': 'What is the day of the month? This should be a number between 1 and 31.'},
{'task': 'T2', 'task_label': 'What is the month?', 'value': 'June'},
{'task': 'T3', 'value': '1929', 'task_label': 'What is the year?'}]
![Page 24: Accessibility for data analysis: using digital tools to unlock archives … · 2020. 1. 17. · using digital tools to unlock archives potential Kathryn Dan, ANU and Ingrid Mason,](https://reader035.fdocuments.us/reader035/viewer/2022071011/5fc8dd161bb26b70ea04bd78/html5/thumbnails/24.jpg)
24
![Page 25: Accessibility for data analysis: using digital tools to unlock archives … · 2020. 1. 17. · using digital tools to unlock archives potential Kathryn Dan, ANU and Ingrid Mason,](https://reader035.fdocuments.us/reader035/viewer/2022071011/5fc8dd161bb26b70ea04bd78/html5/thumbnails/25.jpg)
25
![Page 26: Accessibility for data analysis: using digital tools to unlock archives … · 2020. 1. 17. · using digital tools to unlock archives potential Kathryn Dan, ANU and Ingrid Mason,](https://reader035.fdocuments.us/reader035/viewer/2022071011/5fc8dd161bb26b70ea04bd78/html5/thumbnails/26.jpg)
Example 1901'1901-01-07': 3,
'1901-01-18': 4,
'1901-01-23': 0, # Death of the Queen business abandoned https://trove.
nla.gov.au/newspaper/article/14371864/1343690
'1901-02-25': 4,
'1901-03-18': 0,
'1901-03-29': 0, # missing
'1901-04-04': 3, # No afternoon, day before Easter
'1901-04-09': 0, # Extra Easter Tuesday
'1901-04-10': 0, # Extra Easter Wednesday
'1901-05-27': 0, # Holiday Duke of Cornwall visiting
'1901-05-28': 0, # Holiday Duke of Cornwall visiting
'1901-07-03': 0, # Holiday for polling day
'1901-09-16': 4, # No morning
26
![Page 27: Accessibility for data analysis: using digital tools to unlock archives … · 2020. 1. 17. · using digital tools to unlock archives potential Kathryn Dan, ANU and Ingrid Mason,](https://reader035.fdocuments.us/reader035/viewer/2022071011/5fc8dd161bb26b70ea04bd78/html5/thumbnails/27.jpg)
27
![Page 28: Accessibility for data analysis: using digital tools to unlock archives … · 2020. 1. 17. · using digital tools to unlock archives potential Kathryn Dan, ANU and Ingrid Mason,](https://reader035.fdocuments.us/reader035/viewer/2022071011/5fc8dd161bb26b70ea04bd78/html5/thumbnails/28.jpg)
Contacts and resources• Tim Sherratt https://timsherratt.org/ jupyter
notebook https://nbviewer.jupyter.org/github/wragge/sydney-stock-exchange/blob/master/summary.ipynb
• ANU Archives http://archives.anu.edu.au/
• Digitised registers on Open Research https://openresearch-repository.anu.edu.au/handle/1885/146659
28