Research Design in Modelling & Simulation · Research Design in Modelling & Simulation Instructor...
5
Research Design in Modelling & Simulation Instructor Contact: Instructor: PK Douglas, PhD Office: Partnership II, Room 322 Email: [email protected] Office Hours/Web Hours: After class, or by appointment Course Description: This course provides an overview to modelling and simulation research techniques. This is a core course for graduate students in the Modelling and Simulation department, though it is open to students from other disciplines. It is designed to support development of skills related to machine learning and pattern classification of 4 dimensional data (x,y,z, time), as well as dynamic systems modeling approaches to study the brain. Learning Objectives: Develop the ability to 1) identify important and novel lines of inquiry in the field of Modeling and Simulation research, 2) appreciate how to frame a research problem with the appropriate methodology and experimental design, 3) understand how to implement both hypothesis-driven and model free analytic approaches to study data, 4) recognize the limitations of different research approaches, and determine appropriate conclusions within these constraints. A key theme of the course will focus on identifying alternate model hypotheses, and methods for model selection. Prerequisites: Although there are no formal prerequisites, a background knowledge of linear algebra will be useful, and the ability to code (e.g., Matlab, R, Python) would be helpful for the second half of the course. I am aware that students in the M&S department come from a variety of backgrounds. I will therefore provide tutorials, links, and sufficient time to get up to speed on maths and relevant coding prior to homework assignments. Required Text: There is no required text for this course. Instead, readings will be assigned from the literature and selections from several texts, which will be made available online. Students may find the introductory ML book “Introduction to Machine Learning,” 3 rd . Ed” by Ethem Alpaydin to be a useful reference for that portion of the course. Course Outline: The course is divided into four main parts. In part 1, you will learn the fundamentals of research including topic formulation, experimental design, and classic data types. In part 2 (Model --> Data), you will learn methods for going from a model (hypothesis/hypotheses) to testing how well that model fits the data, compared to alternative models. In part 3, (Data --> Model), you will learn methods for allowing the data themselves to nominate the model using a number of unsupervised and clustering techniques. In part 4, we will discuss methods for critically evaluating these models. Part I: Experimental Design Module 1: Introduction to Research Design 1 HW1: Please take the Pretest—This will fulfill the student engagement requirement for the university. Note – completion of this quiz will receive full credit (5pts). The purpose of this quiz is for me to better understand the students’ backgrounds in the course.
Transcript of Research Design in Modelling & Simulation · Research Design in Modelling & Simulation Instructor...
Research Design in Modelling & Simulation Instructor Contact: Instructor: PK Douglas, PhD Office: Partnership II, Room 322 Email: [email protected] Office Hours/Web Hours: After class, or by appointment Course Description: Thiscourseprovidesanoverviewtomodellingandsimulationresearchtechniques.ThisisacorecourseforgraduatestudentsintheModellingandSimulationdepartment,thoughitisopentostudentsfromotherdisciplines.Itisdesignedtosupportdevelopmentofskillsrelatedtomachinelearningandpatternclassificationof4dimensionaldata(x,y,z,time),aswellasdynamicsystemsmodelingapproachestostudythebrain.LearningObjectives:Developtheabilityto1)identifyimportantandnovellinesofinquiryinthefieldofModelingandSimulationresearch,2)appreciatehowtoframearesearchproblemwiththeappropriatemethodologyandexperimentaldesign,3)understandhowtoimplementbothhypothesis-drivenandmodelfreeanalyticapproachestostudydata,4)recognizethelimitationsofdifferentresearchapproaches,anddetermineappropriateconclusionswithintheseconstraints.Akeythemeofthecoursewillfocusonidentifyingalternatemodelhypotheses,andmethodsformodelselection.Prerequisites:Althoughtherearenoformalprerequisites,abackgroundknowledgeoflinearalgebrawillbeuseful,andtheabilitytocode(e.g.,Matlab,R,Python)wouldbehelpfulforthesecondhalfofthecourse.IamawarethatstudentsintheM&Sdepartmentcomefromavarietyofbackgrounds.Iwillthereforeprovidetutorials,links,andsufficienttimetogetuptospeedonmathsandrelevantcodingpriortohomeworkassignments.
RequiredText:Thereisnorequiredtextforthiscourse.Instead,readingswillbeassignedfromtheliteratureandselectionsfromseveraltexts,whichwillbemadeavailableonline.StudentsmayfindtheintroductoryMLbook“IntroductiontoMachineLearning,”3rd.Ed”byEthemAlpaydintobeausefulreferenceforthatportionofthecourse.
Course Outline: Thecourseisdividedintofourmainparts.Inpart1,youwilllearnthefundamentalsofresearchincludingtopicformulation,experimentaldesign,andclassicdatatypes.Inpart2(Model-->Data),youwilllearnmethodsforgoingfromamodel(hypothesis/hypotheses)totestinghowwellthatmodelfitsthedata,comparedtoalternativemodels.Inpart3,(Data-->Model),youwilllearnmethodsforallowingthedatathemselvestonominatethemodelusinganumberofunsupervisedandclusteringtechniques.Inpart4,wewilldiscussmethodsforcriticallyevaluatingthesemodels.PartI:ExperimentalDesignModule1:IntroductiontoResearchDesign1 HW1:PleasetakethePretest—Thiswillfulfillthestudentengagementrequirementfortheuniversity.
Note–completionofthisquizwillreceivefullcredit(5pts).Thepurposeofthisquizisformetobetterunderstandthestudents’backgroundsinthecourse.
Module2:Formulatingandtestinghypotheses1 Stepsinconstructingtestablehypotheses2 CorrelationandCausation(WhatinferencescanImake?)3 HW2:TopicformationandIntroduction.Inafewsentencesinadiscussionthread,pleasedescribe3
topicsthatinterestyouinmodelingandsimulation.Pleasetakethistimetointroduceyourselftotheclassanddescribeyoureducational/workbackground.(5pts)
Module3:Datasources&DataTypes1 Datatypes(descriptive,correlational,etc)2 Potentialthreatstovalidity,generalizability,andreplicability3 PilotTesting&Identifyingpopulationimplicationsaprioriandtheirinfluenceonstatisticalchoices4 HW3:(GroupAssignment)FormingResearchHypotheses—Discussionpostof2researchquestions,
eachwith3testablehypotheses.Responsewith2recommendations,requestsforclarification,and/oradditionalpotentialhypotheses.*Youmustsubmitanemailofwhoyourgroupmembersarebyweek4(groupsofupto3people)(5pts)
Module4:Datacollection&Design1 Quantitative(e.g.datatypes,computationaltechniques,physiological)2 SamplingandStatistics3 HW4:(GroupAssignment)—Discussionpost.Postadiagramofyourresearchdesign.IncludeDVsand
IVs,interventions/manipulations.Useyourdiagramtoshowhowinformationcollectedwillhelpansweryourresearchquestionbytestingyourhypotheses.Doyouplantouseanexistingdatasetorasimulateddatasetforyourgroupproject?(10pts)
PartII:ModelHypothesis-->DataModule5:ClassicalInference1 ClassicalHypothesisTesting2 GeneralLinearModel3 HW5:GLMexercise(5pts)Module6:CorrectionforMultipleComparisons1 Falsediscoveryrate(e.g.,Benjamini-Hockberg)2 Othermethodsforcorrectionformultiplecomparison3 HW6:WatchthevideoonGaussianRandomFieldtheoryandcompletethequiz.(5pts)Module7:InductiveBias&CorrectionforMultipleComparison1 InductiveBiasInDataModeling&Simulation2 FeatureSelection,Cross-validation,permutationtests3 HW7:Machinelearningexercise(5pts)
Module8:MachineLearning-PartI1 FeatureSelection-Isthisstepimportant?2 DimensionReduction(CurseofDimensionality,KernelTrick)3 HW8:DownloadWeka,andcompletethefeatureselectionexercises(5pts)Module9:MachineLearning-PartII1 InterpretingClassifiers
2 Hownoiseandredundancycaneffectinterpretation3 HW9:CompleteMatlabLDAassignment(10pts)Module10:GenerativeModels1 DynamicSystemsModeling2 ModelInversion(ParameterEstimationTechniques)3 HW10:CompleteMatlabODEassignment(10pts)Module11:FrequencyDomainAnalysis1 LaplaceTransform2 StabilityconstraintswhenmodelingintheFrequencyDomain3 HW11:CompleteFrequencyDomainAssignment(5pts)PartIII:Data-->ModelModule12:UnsupervisedMethods1 Whyrunanunsupervisedanalysis?2 ReviewofLinearAlgebraTechniques3 HW12:CompleteLinearAlgebraReview(5pts)Module13:EigenvaluesandMore1 SingularValueDecomposition(SVD,PCA,ICA,CCA)2 HW13:CompleteMatlabAssignment(10pts)Module14:ModelSelection4 Sparsevs.ComplexModels5 AIC,BIC,andFreeEnergy6 HW11:CompleteModelSelectionExercise(5pts)PartIV:EvaluatingModelModule15:CritiquingModels1 Understandinglimitationsofmodels2 HW15:Readpostedmanuscriptandpostdiscussionoflimitationsofthemethodinthediscussion
section.(10pts)Module16:GroupWeek1 Workinyourgroupsonyourproject
Course Requirements: Attendance,participation,completionofhomeworkandfinalexamarerequiredforcoursecompletion.Accordingtotheuniversity,thisclassis"mixedmode".Hereistheofficialdescription:M–MixedMode/ReducedSeatTimecoursesincludebothrequiredclassroomattendanceandonlineinstruction.Classeshavesubstantialactivityconductedovertheweb,whichsubstitutesforsomeclassroommeetings.
Evaluation & Grading: Homeworkandparticipationinthediscussiontopicswillaccountfor100ptsofyourgrade.Thefinalprojectwillalsobeworth100pts(oralpresenation,50pts,andfinalpaper50pts).Theclasswillfollow
atraditionalgradingscaleshownbelow:LetterGrade PointsA 90-100B 80-89C 70-79D 60-69F 59orbelow
Make up Exams: Homeworks,groupprojects,andexamswillbepostedonline,andstudentsmaytakethemanytimeupuntiltheirfinalduedate.Allgroupmembersareexpectedtocontributetothebestoftheirability,andareexpectedtoparticipateinthefinalgrouppresentationandcontributetothefinalpaper.Groupsshouldsubmitfinalpapersinresearchmanuscriptformat,andindicatethecontributionofeachgroupmember.Ifforsomereason,youcannotmakeitin-persontothefinalgrouppresentation,pleasecontactmetodiscussalternatives.Failuretodosowillresultinalowergradeforthecourse. Attendance Policy: Accordingtotheuniversity,thisclassis"mixedmode".Hereistheofficialdescription:M–MixedMode/ReducedSeatTimecoursesincludebothrequiredclassroomattendanceandonlineinstruction.Classeshavesubstantialactivityconductedovertheweb,whichsubstitutesforsomeclassroommeetings.WhileIdonotdirectlypenalizeformissedclasses,attendanceitisextremelyimportantforsuccessandclassparticipation,andshouldbeconsideredmandatory. Academic Engagement: Allinstructors/facultyarerequiredtodocumentstudents’academicactivityatthebeginningofeachcourse.Inordertodocumentthatyoubeganthiscourse,pleasecompletethepretestassoonaspossible.Failuretodosomayresultinadelayinthedisbursementofyourfinancialaid.Todocumentyouracademicengagement,youwillcompleteaquizbytheendofthefirstweekofclass. Final Paper: ThefinalpaperwillbedueatthetimeassignedbyUCF.ThistimewillbepostedonWebcourseswithinthefirstweek.
Changes/Announcements: Itispossiblethatadjustmentstotheschedulemaybemadeduringthecourse.Intheeventthatanythinginthissyllabuschanges(e.g.classroommoves,changesinduedates,contactinformation),Iwilluseabroadannouncementsothatallstudentswillbeinformedimmediately.Itiscriticallyimportantthatyousetyourwebcourseannouncementssothatyoureceiveallnotificationsandbesuretochecktheclasswebsiteregularly. Academic Integrity: Studentsareencouragedtodiscussproblemswithcolleagues,butthefinalassignmenthandedinshouldbethestudent’sownwork.Plagiarismandcheatingofanykindonanexamination,quiz,orassignmentwillresultatleastinan"F"forthatassignment(andmay,dependingontheseverityofthecase,leadto
an"F"fortheentirecourse)andmaybesubjecttoappropriatereferraltotheOfficeofStudentConductforfurtheraction.SeetheUCFGoldenRuleforfurtherinformation.Iwillalsoadheretothehigheststandardsofacademicintegrity,sopleasedonotaskmetochangeyourgrade. Accessibility Statement: TheUniversityiscommittedtoprovidingreasonableaccommodationsforallpersonswithdisabilities.Thissyllabusisavailableinalternateformatsuponrequest.Studentswithdisabilitieswhoneedaccommodationsinthiscoursemustcontacttheprofessoratthebeginningofthesemestertodiscussneededaccommodations.Noaccommodationswillbeprovideduntilthestudenthasmetwiththeprofessortorequestaccommodations.StudentswhoneedaccommodationsmustberegisteredwithStudentAccessibilityServices,FerrellCommons,7F,Room185,phone(407)823-2371,TTY/TDDonlyphone(407)823-2116,beforerequestingaccommodationsfromtheprofessor. Copyright: Thiscoursemaycontaincopyrightprotectedmaterialssuchasaudioorvideoclips,images,textmaterials,etc.TheseitemsarebeingusedwithregardtotheFairUsedoctrineinordertoenhancethelearningenvironment.Pleasedonotcopy,duplicate,downloadordistributetheseitems.Theuseofthesematerialsisstrictlyreservedforthisonlineclassroomenvironmentandyouruseonly.Allcopyrightmaterialsarecreditedtothecopyrightholder.
