Agent Development¶
Agents are responsible for driving the execution of the GUI application. Therefore, their implementation vary according to the Use Case they are addressing.
The following chapter introduces a simple example of an Agent which interacts with every available object in a Windows application at most once.
Preparing the Interpreter¶
The Interpreter is responsible for constructing the GUI State and its available Actions. Interpreters offer a generic interface but their implementation depends on the Automation components which are provided. In this example, the libvirt and vnc based Automation interface will be used. Their details can be found in murphy/automation.
First, the Feedback and Control interfaces are built using their factory classes.
from murphy.automation.control import LibvirtControl
from murphy.automation.feedback import LibvirtFeedback
domain_id = "libvirt-vm-name" # libvirt Domain ID, name or UUID
vnc_socket = "localhost:5900" # VNC connection socket or address
control = LibvirtControl(vnc_socket, domain_id)
feedback = LibvirtFeedback(vnc_socket, domain_id)
Secondly, the Windows scraper client is initialized. Mr. Murphy relies on a scraping agent running within the guest OS in order to de-construct the GUI content of a Windows application.
from murphy.model.scrapers.uiauto import WinUIAutomationScraper
scraper_address = "192.168.22.32" # address of the scraper agent within the guest OS
scraper_address = 8000 # port of the scraper agent within the guest OS
scraper = WinUIAutomationScraper(scraper_address, scraper_port)
Lastly, the Windows Interpreter class is constructed using the above interfaces.
from murphy.model.interpreters.windows import WindowsInterpreter
interpreter = WindowsInterpreter(feedback, control, scraper)
The Travel Journal¶
The Travel Journal provides useful information regarding the execution of the GUI application.
from murphy.journal import Journal
journal_path = '.' # local path where to store Journal information
journal = Journal(journal_path)
Exploring the application¶
Once all the resources are initialized, the control can be handed over to the main loop which implements the application exploration logic. The logic can be summarised as follows:
- Interpret the current visible state of the GUI application
- Compare the given state with the known ones in the Journal
- Select a new action to perform
- Perform the action
- Return to point #1
import time
while True:
new_state = interpreter.interpret_state()
node = update_journal(new_state)
action = choose_action()
action.perform()
time.sleep(3)
Once a new state is returned by the Interpreter, the logic compares it with the Journal content. The Journal will return an old node or a new one according to whether the new state was ever encountered before.
The update_journal function sets the new node as the current one and renders the Journal as an HTML page.
def update_journal(state):
if state in journal:
node = journal.find_node(state)
else:
node = journal.new_node(state)
journal.current_node = node
journal.render()
return node
The application State the Node refers to can be accessed from its namesake attribute. The State encapsulates the available Actions which can be performed. In order to acknowledge whether the Action was already performed, a boolean attribute is appended to its instance.
The action to perform is randomly chosen among the ones which have not been performed yet.
import random
def choose_action(node):
actions = []
for action in node.state.actions:
if not hasattr(action, 'performed'):
action.performed = False
actions.append(action)
elif not action.performed:
actions.append(action)
action = random.choose(actions)
action.performed = True
return action
Conclusions and references¶
The above implemented Agent is intended to be simple to give a complete yet easy to understand illustration of Mr. Murphy abstraction layers and their integration.
Several aspects such as the handling of errors and corner cases where purposedly omitted. This Agent implementation will wander aimlessly within a GUI application until it finds a State with no available Actions to perform and then it will crash.
More complete examples can be found in the folder murphy/agents.