A Method for Evaluating End-User Development Technologies
-
Upload
claudia-melo -
Category
Technology
-
view
164 -
download
0
Transcript of A Method for Evaluating End-User Development Technologies
Americas Conference on Information Systems,
AMCIS 2017, Aug, 10th
A METHOD FOR EVALUATING END-USER DEVELOPMENT TECHNOLOGIES
Prof. Dr. Claudia Melo, Dep. of Computer Science Jonathan Moraes, Fac. of Software Engineering Marcelo Ferreira, Fac. of Software Engineering
Prof. Dr. Rejane Figueiredo, Fac. of Software Engineering University of Brasília, Brazil
WHY ARE WE INVESTIGATING EUD TECHNOLOGY EVALUATION?
• # of end users > # of professional so1ware developers • > 50M end-user developers,
USA only (Burne9 and Myers, 2014)
• Li9le research has empirically examined so1ware package evalua7on criteria and techniques in general, and in the end-user development context in par7cular (Harnisch, 2014; Jadhav and Sonar, 2009; Jadhav and Sonar, 2011; Misra and Mohanty, 2003)
• Success and failure of end-user development within an organizaOon ulOmately depends on how effec7ve so1ware packages are chosen and used
2
• This research study is part of a partnership with the Ministry of CommunicaOons, Science, and Technology, Brazil
• Plenty of EUD tools being offered in the market
• OrganizaOons sOll lost on how to take full advantage of EUD (facing cultural, managerial and technological challenges). But already buying soluOons!
INDUSTRY CONTEXT
3
Three different areas have important, but parOal, contribuOons to our research purpose:
•So#ware package acquisi1on research;
•So#ware quality models & CSCW/HCI research, and
•Technology acceptance research.
LITERATURE REVIEW
4
WHAT WAS CONSIDERED AS A STARTING POINT?
The model should evaluate technologies that have: •Essen1al quali1es that enable the end-user developer to manipulate the
tool and produce useful results in a certain applica1on domain •from so]ware quality models and CSCW/HCI models;
•General quali1es inherent to so#ware packages •from so]ware package acquisiOon models and technology acceptance models;
•Essen1al quali1es for management and technological governance •from so]ware package acquisiOon models;
•An evalua1on method based on already-established and tested techniques, even if they come from a different context
5
• Evalua1on criteria, characteris1cs, sub characteris1cs, and aJributes
• Developed ques1ons and metrics from different points-of-view (PoV).
STRUCTURING THE EVALUATION MODEL
6
THE EVALUATION METHOD
1. Functional characteristics:
○Functionality (3 questions, PoV: Governance/Platform)
○Collaboration (13 questions, PoV: EUD/Platform)
○Data Management (4 questions, PoV: EUD/Platform & 4 questions, PoV: Governance/Platform)
2. Cost and Benefit Characteristics (6 questions, PoV: Governance/Platform)
3.Vendor Characteristics (4 questions, PoV: Governance/Platform)
4. Software Quality Characteristics
○ Compatibility (6 questions, PoV: EUD / Platform)
○ Maintainability (6 questions, PoV: EUD / Platform)
○ Usability (212 questions, PoV: EUD / Platform)
○ Reliability (7 questions, PoV: IT Manager/Platform)
○ Performance Efficiency (6 questions, PoV: EUD / Platform)
○ Security (5 questions, PoV: Governance/Platform & 1 question, PoV: EUD/Platform/Developed Application & 9 questions, PoV: EUD / Platform)
5.Hardware & Software Configuration (8 questions)
The complete EUD technology evaluation model is available at: https://itrac.github.io/eud_technology_evaluation7
EVALUATING EUD TECHNOLOGIES - STEPS
1. Determining the need, including high-level investigation of software features and capabilities provided by vendors August/2016 to October/2016 - literature review and contact with leaders of public and private organizations to build a general list of tools.
2. Short listing candidate packages and eliminating the candidate packages that do not have the required feature We shortlisted the most solid market offers (Forrester, 2016).
8
3. Using the proposed evaluation technique to evaluate remaining packages and obtain a score
4. Pilot testing the tool in an appropriate environment (parallel to 3) Fundamental to refine the model proposed (removing, rewriting, and adding questions/metrics).
The evaluation model and the platform evaluation results presented in this work are already the result from a second evaluation iteration
EVALUATING EUD TECHNOLOGIES - RESEARCH STEPS
9
EXECUTION DETAILS
The platforms:
● OutSystems
● Oracle Apex
● Salesforce Lightning¹
● Zoho Creator¹
1 - The Salesforce Lightning and Zoho Creator evaluations are not described in the paper.
10
Testing:
• 3 software engineering students as testers;
• Each student performed the entire model for every platform using four default scenarios (C-R-U-D)
• Either using the platform’s predefined templates or not.
• A researcher supervised the testers and validated the forms.
EXECUTION DETAILS
11
Results: Functional Characteristics
Functionality
Sub-Characteristics Oracle Apex OutSystems Salesforce Lightning Zoho Creator
Application Domain Database General-purpose General-purpose Database
Collaboration
Shareability ✔ ✔ ✔ ✔
Coordination of Actions ✔ ✔ ✔ ✘
Consequential Communication
60% 60% 60% 80%
12
Results: Functional CharacteristicsCollaboration
Sub-Characteristics Oracle Apex OutSystems Salesforce Lightning Zoho Creator
Finding Collaborators and Establishing Contact
75% 0% 50% 100%
Concurrent Protection 0% 50% 0% 50%
Data Management
Data Input and Output 100% 100% 100% 66%
Required Technical Knowledge
33% 33% 33% 0%
13
Results: Cost and Benefit & Vendor Characteristic
Cost
Sub-Characteristics Oracle Apex OutSystems Salesforce Lightning Zoho Creator
License Cost $ 164,839.00 $ 2,072,601.74 $ 3,600.00 per user/year
$ 1999.00 per month
Maintenance Cost N/A N/A N/A N/A
Vendor
Contract Dependency 100% 100% 100% 100%
Technology Dependency
100% 0% 0% 100%
14
Results: Software Quality Characteristics
Compatibility
Sub-Characteristics Oracle Apex OutSystems Salesforce Lightning Zoho Creator
Technical Knowledge Requirement
Advanced Advanced N/A Average
Data Exchangeability ✘ ✔ ✘ ✘
Connectivity With External Component/System
Possible Possible Impossible Possible
Reusability ✔ ✔ N/A ✔
15
Results: Software Quality Characteristics
Maintainability
Sub-Characteristics Oracle Apex OutSystems Salesforce Lightning Zoho Creator
Modifiability 100% 100% 50% 100%
Reusability Possible, Easy Possible, Easy Possible, Average Possible, Hard
Reliability
Availability N/A N/A N/A N/A
Vendor Support 0.41 fixes/day 0.43 fixes/day 0.12 fixes/day 0.73 fixes/day
16
Results: Software Quality Characteristics
Performance Efficiency
Sub-Characteristics Oracle Apex OutSystems Salesforce Lightning Zoho Creator
Response Time 100% 100% 100% 100%
Turnaround Time 50% 100% 75% 100%
Security
Access Behaviors 80% 60% 60% 20%
Developed Application Security
100% 100% 100% 0%
17
Results: Software Quality Characteristics
Security
Sub-Characteristics Oracle Apex OutSystems Salesforce Lightning Zoho Creator
Update Behaviors 100% 50% 100% N/A
File Upload Security 100% 100% 100% 100%
Report Behaviors ✔ ✔ ✘ ✘
Security Algorithms 50% 100% 100% 50%
18
CONCLUSION
● The major original contributions of the paper are:
○ A detailed method for evaluating EUD technologies that comprises 11 characteristics, 20 sub-characteristics, 30 attributes, 300 questions/metrics, and
○ Evaluations using the method against 4 leading EUD platforms in the market (we described only 2 in the paper)
○ Next steps: validation with real-world scenarios (e.g. action research) & investigation of automation opportunities
20
Questions? Thanks!
Claudia Melo21
Acknowledgement: this research received support from
Fundação de Apoio a Pesquisa do Distrito Federal (FAP-DF)