Refactoring(and testing, and design, …)

Maurício Anichem.finavaroaniche@tudelft.nlDelft University of Technology2019

Agile is not only about processes…

• Technical practices are also very important:• Automated testing• Test-Driven Development• Pair programming• Refactoring• Continuous Integration

The clean code game

• We are now responsible for improving the pacman game.• Inspect the source code at

http://bit.ly/ugly-pacman.• What would you improve in there?

Come up with a list of problems and how you would fix them.

• You have 10 minutes to understand and propose improvements.

A long list of problems…

• Duplicated code• Long functions• No modularity• Global variables• Lack of clarity• Untestable (lack of controllability / observability)

Duplicated code

Long functions

• The implementation is currently a single ‘spaghetti’.• 225 lines!

• Hard to understand• No reuse.

No modularity

• The code has different responsibilities.• Functional perspective:• Map• Characters• Game

• Separation of concerns:• UI (user interface)• The game itself

Global variables

• “map”, “game_finished”, “win” are global variables.• Actually, all variables there are global as there are no functions…

• Global variables are a perfect place for bugs.

Lack of clarityWhat do these pieces of code do?

Untestable

• Can you write automated tests for this program?

• There are no clear, concise units to be tested.• No easy way to control/observe:• UI prints on the screen, reads from keyboard• Random number generation

“Continuous attention to technical excellence and good design enhances agility.”

-- The Agile Manifesto

Refactoring

• Refactoring (noun): a change made to the internal structure of software to make it easier to understand and cheaper to modify without changing its observable behavior.• Refactoring (verb): to restructure

software by applying a series of refactorings without changing its observable behavior.

Martin Fowler

Long list of known refactoring

• Extract method• Introduce variable• Move method• Rename variable/function• Decompose conditional• Remove parameter• …• See full catalog at https://refactoring.com/catalog/

Let’s now explore a refactored version!

• Explore the code in this repository: https://github.com/mauricioaniche/sem-pacman-python/tree/master/refactored• It is the same game. But now the code is

definitely better!• What is better? Make a list.• What can be improved? Make a list.

You have 10 minutes!

Separate UI logic from the rest

• See the ui.py file. It contains all the UI logic.• Only knows the game’s data structures• Like ‘map’• And the characters in a map (G, @, …)

• Pacman.py doesn’t contain a single call to the UI.• It only contains the business logic related to the game.

• Program.py coordinates the flow between UI and Pacman modules.• One might call it “Controller”.

• Additional advantage: you can change the UI.• We’ll understand how soon.

Small functions• The single spaghetti method was broken into many small functions.• See total_pills() and find_pacman().• They are easily testable now.

Extracting common logic

• See within_borders(), is_wall(), is_ghost(), is_pill().

• They have similarities…

Extracting common logic

• We may move the common logic to a single method.

This could be moved to a separate method.

• Simpler to understand

• Easier to test

Small method!

No global variables

• “map”, “win”, and “game_finished” are now local variables.• Smaller scope, bigger control.

• Note that having “map” as a parameter allows us to:• Easily change the map that will be played.• Facilitates testing.

Refactoring for testability• UI and random generation are

encapsulated inside functions that we control.• We pass these functions as parameters

to run().• We may call it “Dependency Inversion”.

• Why?• Ease controllability: we now can simulate

keyboard and random numbers.• We call these simulations, “mocks”.• See program_integration_tests.py,

functions fake_ui_key() and fake_int().• See pacman#move_ghosts() method.

Lots and lots of tests

• 100% statement coverage in pacman.py module.• Boundary testing.• I actually found a bug…

Bug: <= where it should be <

System tests

• Some end-to-end tests are also possible, given that we have more controllability.• Observability: our run()

method returns the result of the game.• Note that move_ghosts() and

play() also return values, mostly for observability.

Avoiding flaky tests

• Look at pacman_tests.py.• What would happen if “map” was

not created for every single test in there? In other words, if we had a global variable map?• The test would be flaky!• Try it!

Design, Tests, Quality…

• Note how design, tests, and quality are related to each other.• Good design => easier to test.• Refactoring => good design.

• One enables the other.

Low- and High-level refactoring

• Low-level refactoring:• Happen at function level, maybe at class level.• Example: extract method, rename variable.

• High-level refactoring:• Design changes• Example: move UI rules to a different module.

• We just did both.

Challenges for you!

• Different ghost walking strategies.• Maybe these strategies deserve to be in a dedicated module?

• Implement a special pill that freezes ghosts for two rounds.• Write unit tests for it!

• Explore the OO solution.

• Found more improvements to do? Open a pull request!! J• https://github.com/mauricioaniche/sem-pacman-python/

Code smells

• “A code smell is a surface indication that usually corresponds to a deeper problem in the system. The term was first coined by Kent Beck while helping me with my Refactoring book.”

Martin Fowler (https://martinfowler.com/bliki/CodeSmell.html)

• Examples of code smells are:• Long methods• Feature envy• Shotgun surgery• …

Code smells are bad…• Tufano et al [1] after an empirical study in more than 200 open source

projects, showed that most code smells are introduced during the creation of the file (and not introduced during maintenance phase, as one could suppose).• Khomh et al [2] and Li and Shatnawi [3] showed that smelly classes are more

prone to change and to defects than other classes. • Palomba et al [4] showed that smells related to complex or long source code

are perceived as harmful by developers; other types of smells are only perceived when their intensity is high.• Moonen et al [5] conducted a survey with 85 professionals, and results

indicate that 32% of developers do not know or have limited knowledge about code smells.

References• [1] Tufano, Michele, et al. "When and why your code starts to smell bad." Proceedings of

the 37th International Conference on Software Engineering-Volume 1. IEEE Press, 2015.• [2] Khomh, Foutse, et al. "An exploratory study of the impact of antipatterns on class

change-and fault-proneness." Empirical Software Engineering 17.3 (2012): 243-275.• [3] Li, Wei, and Raed Shatnawi. "An empirical study of the bad smells and class error

probability in the post-release object-oriented system evolution." Journal of systems and software 80.7 (2007): 1120-1128.• [4] Palomba, Fabio, et al. "Do they really smell bad? a study on developers' perception of

bad code smells." Software maintenance and evolution (ICSME), 2014 IEEE international conference on. IEEE, 2014.• [5] Yamashita, Aiko, and Leon Moonen. "Do developers care about code smells? an

exploratory survey." Reverse Engineering (WCRE), 2013 20th Working Conference on. IEEE, 2013.

Static versus Dynamic Testing

Static Testing: Testing of a component or system

at specification or implementation level without execution of software (e.g., reviews or static analysis)

Dynamic Testing:Testing that involves the execution

of the software of a component or system.

Static analysis tools

• They help developers in automatically detecting several problems.• Examples:• Uninitialized variables• Unreachable code• Lack of comments• Long methods

• A huge research field!

Pylint• Finds common problems

in our code automatically.

• Lots of them: http://pylint.pycqa.org/en/latest/technical_reference/features.html

Pylint messages

Extracted from Pylint user guide: https://media.readthedocs.org/pdf/pylint/latest/pylint.pdf

Refactoring in legacy systems?

• The problem with legacy code is that it’s not testable.• Legacy management strategy:

1. Identify change points2. Find an inflection point.3. Cover the inflection point.

1. Break external dependencies2. Break internal dependencies3. Write tests

4. Make changes5. Refactor the covered code.

• In other words, write tests for it before breaking it into small testable units.

Summary: http://www.netobjectives.com/system/files/WorkingEffectivelyWithLegacyCode.pdf

License

• You can use and share any of my material (lecture slides, website).• You always have to give credits to the original author.• You agree not to sell it or make profit in any way with this.

• Material that I refer has its own license. Please check it out.

Images in this presentation

• Pacman, by hobermallow: https://openclipart.org/detail/176294/pacman-3d• Aquarela, by valessiobrito:

https://openclipart.org/detail/4421/aquarela-colors• Sad face, by technaturally:

https://openclipart.org/detail/279690/sad-face-emoticon-yellow