2006 Technical Report

Deb Polson and Yang Wong

The SCOOT SMS System was used by SCOOT in several ways:

  • Register and verify team’s details before and during the event
  • Create entire game puzzle path (includes scoring and custom result handling)
  • Process solution attempts by the team (in response to puzzles) and respond appropriately (send hint/incorrect or next puzzle)
  • Prompt third party SMS services to send and receive all SMS to and from teams during the event
  • Update scores when successfully completing puzzles
  • Provided a means to easily administrate (modify, view, add) gameplay data (current path progression, SMS traffic) for technical support during the event
  • Records as much gameplay data as possible for post analysis
  • Custom Result Handling: Passes on SCOOT specific data to the SCOOT System (quest logging, inventory)

Resulting statistics of rego and play

A. How many teams, started and completed in each site.

Started Completed Percentage
Australian Centre for the Moving Image (ACMI)
16th 35 19 56%
17th 44 35 74%
18th 44 38 83%
Total 123 92 72%
The Arts Centre (AC)
16th 14 10 71%
17th 24 19 79%
18th 27 21 78%
Total 65 50 77%
State Library of Victoria (SLV)
16th 14 13 93%
17th 10 9 90%
18th 25 19 76%
Total 49 41 84%
Melbourne Museum (MM)
16th 10 8 80%
17th 19 13 68%
18th 19 19 100%
Total 48 40 83%

B. How many teams played (at least started), at how many sites, during how many days

1 2 3 4 Total Sites Started
1 72 41 7 7 127
2 1 4 1 16 22
3 0 0 0 3 3
Total 73 45 8 26
Days

C. How many messages sent before and during the event and related cost totals

Service Used Messages Sent Setup Cost ($) SMS Cost ($) Total Cost ($)
Traitel 1515 121.00 149.99 270.99
MessageNet 2351 395.00 0 395.00
Total 3866 516.00 149.99 665.99

The number of messages received is always the same as the number of messages sent. Totals include messages sent for testing purposes before and during the event.

D. How long did it take to complete, each site, each day ( Average / Minimum / Maximum )

Day ACMI AC SLV MM
16th 122 / 80 / 210 53 / 40 / 90 71 / 40 / 100 112 / 50 / 180
17th 100 / 50 / 150 66 / 25 / 110 58 / 40 / 70 61 / 30 / 100
18th 54 / 40 / 110 48 / 25 / 90 50 / 30 / 80 63 / 30 / 120

At ACMI on the 18th the NGV game path (5 puzzles) was removed thus significantly reducing the completion time. 1 puzzle was removed within the SLV game path on the 17th and 18th. Most solution issues for puzzles were resolved by the middle of the 17th and all resolved by the start of the 18th. The SMS service temporarily ceased during the 17th thus throwing out of sync around 10 teams overall. The above statistics were obtained from figures rounded off to the nearest 5 minute interval.

E. How many teams registered before and during the event

310

F. How many teams logged into SCOOT World during the event

243

SCOOT developers report on  specific components

A. Development

Integration with External Applications and APIs

The SCOOT SMS System has been created so that events/games like SCOOT can make use of its SMS logic engine without having to delve deep into its implementation. As a result, most of SCOOT’s non-game path related data was kept separate to SCOOT SMS System. This separation of systems is important as it allows for greater portability and independence. For example, SCOOT’s Virtual World can be played nearly in its entirety without the SCOOT SMS System being active (scoring is tied to the SMS system, but can be completely separate if necessary) and the SMS ‘treasure hunt’ component of SCOOT can be played in its entirety without SCOOT World.

SMS APIs offered by SMS service provider Traitel were easily integrated and could be enabled/disabled quickly without much hassle.

Game Path Creation

SCOOT’s game path was technically entirely created using the Advanced Game Pathing interface of SMS SYSTEM Development and Administration Tools. The interface allowed developers to specify:

  • The details of each puzzle (clue, solutions, hints unused)
  • The sequence in which puzzles were delivered to teams by SMS
  • The scores associated with completing a puzzle or string of puzzles
  • Custom results that are sent to the SCOOT system for processing (quest logging, inventory)

The logical sequencing of puzzles/SMS proved to be very useful in overcoming particular physical site issues during the event (NGV Australia being closed Monday).

The SCOOT game path was specified, reviewed and modified many times during the development process. Below are screenshots of the final game path diagram that was used for the event.

B. Event Setup and Support

The SCOOT event used two SMS service providers to enable SMS to be sent to and received from teams. These are:

  • MessageNet – http://www.messagenet.com.au/
  • Traitel Telecommunications – http://www.traitel.com.au/

Setup for both services was initially straightforward and both functioned well during development and testing but during the event itself, issues arose that had varying impact on the operation of the event.

SMS Setup Issues

Initially it was proposed that SCOOT be designed, and therefore budgeted, for 300 groups in total to register and play during the event. It was intended that ACMI would acquire their own MessageNet account with a dedicated mobile number for the event (as they had previously worked successfully with the SCOOT  events in the past). This meant they were responsible for payment and set up.

However, in August (very close to the actual event!) the ACMI representative increased the number to over 900. The MessageNet account was not going to be feasible with these numbers, as the budget could not support it.  So the SCOOT team was able to quickly find alternative providers. Triatel was decided upon for reasons of cost, access and ease of use.

(PLEASE NOTE: that exactly 310 groups registered as we expected. For the previous 8 months of development, the number of 300 was considered as there were both predictable and unpredictable implications on the hosting of the event – physically, financially, technically and in terms of support personnel. Most significant is the ‘physical’ component to the game. We had to accommodate for the ‘nature’ of the sites, such as the library, when designing the activities and pathways of the players. Also, as SCOOT is simply a linear path, we wanted to avoid crowding at the interactive nodes as this would delay groups and take away from their sense of discovery.)

MessageNet (SCOOT account)

Since no account was set up for ACMI due to the increase in numbers expected to play. The SCOOT account was planned (and therefore pre-programmed into the system) to be a backup service provider if needed during the event. No additional setup was required for MessageNet.

Traitel (ACMI account)

The SCOOT event decided to use Traitel as they offered significantly lower SMS costs than MessageNet and other similar competitors. They also offered services, like a dedicated phone number, detailed SMS tracking and SMS data redirection, which are critical/very useful to the operation of the event. Traitel was also hoped to allow all types of mobile phone plans to work with the SMS System (specific Vodafone networks don’t work with MessageNet).

In the case of Traitel, an initial deposit had to be made before certain services could be activated and this is where issues of billing arose. Traitel have two methods of payment, credit card or BPay, but due to the delay associated with BPay, a credit card was the only option.

The Australian Centre for the Moving Image (ACMI) requires that all organisational credit card details are kept private from anyone other than the director of the organisation. This posed several issues.

The privacy required meant that only a single person could initiate the activation process as credit card details are entered directly into Traitel’s web interface for customer accounts. The availability of such a person was limited thus setup occurred later than expected.

When Traitel requested a scan/fax of the card itself for confirmation, there was an even further delay as the credit card is actually kept in a safe. Unfortunately this confirmation requirement was not indicated on their website until after the initial set of credit card details were entered.

In able to send SMS using Traitel’s API, the account’s username and password is required every single time. This meant that the billing details had to be further protected using another password. Fortunately this was possible through Traitel’s account features.

Traitel required a deposit into the account before being able to send SMS. The number of SMS to be sent during the event is unpredictable. One would be wary of the cost associated with over compensating by depositing for the upper most limit of SMS. Fortunately Traitel provided the ability to automatically deposit a certain amount when a lower limit is reached, but this would still involve unnecessary spending. Fortunately an arrangement was made with Traitel that meant that ACMI could request a refund of unused credit after the event is over. Other service providers may not have been so forthcoming.

Service Issues

Several issues arose during the SCOOT event that impacted heavily on its operation. In the end, all problems were resolved and the event continued on with very limited interruption.

Pre Event

Traitel were informed of the event’s requirements prior to its commencement. This included an estimated amount of SMS traffic to be expected, response time requirements for SMS data and guaranteed uptime of the SMS service.

Day One

During the event, as more teams started sending to and receiving SMS from the dedicated number, an obvious degradation of SMS response time was noticed. Traitel were contacted by email and fortunately they managed to identify and resolve the issue by distributing SMS traffic amongst several mobile numbers. This meant that teams could send SMS to any of these numbers and Traitel would pass on the details to the SMS  System. It is assumed that Traitel didn’t fully realise how that much SMS traffic would impact on the performance of a single dedicated number.

Many attempts to contact Traitel by phone first proved to be unsuccessful. Unlike Day 2 (Sunday 17th September), it seemed Traitel were prepared to quickly and effectively respond to emails. Whether that was because of the information sent to them, prior to the event is unknown. The dedicated number congestion issue was deemed resolved by both Traitel and the SCOOT support/development team. SMS records in the SMS System database indicated that response time had improved dramatically after the issue was resolved.

There were no reports of teams that were completely unable to send and receive SMS to any of Traitel’s numbers.

Day Two

Confidence in Traitel’s service was high but due to Day One’s issues, support staff were ready to conduct fixes the first sign of trouble. The day started well but by about an hour in, degradation of response times could be noticed. In fact it was much worse than Day One as it seemed many sent SMS were not reaching their destinations. Support staff attempted to contact Traitel by phone first, then email, but were unable to get a response. As a result, the SMS component of the SMS System was switched to MessageNet.

SMS sent to either MessageNet or any of Traitel’s numbers were directed to the SMS System This meant that teams from Day One and early Day Two could continue without having to switch numbers. As an added precaution, in case Traitel’s service became completely congested (receiving SMS was still operating well), introductory Quest Logs in the SCOOT Virtual World were edited to reflect the new MessageNet number instead of Traitel’s dedicated number. Staff at all SCOOT Headquarters were also informed of the number change and prompted to provide it as the primary number.

Response times improved considerably with MessageNet handling all outgoing SMS. As anticipated, one team with a Vodafone Rewards phone account could not send to MessageNet. This was resolved by supplying a support staff’s number and the dedicated Traitel number to SMS to. For the rest of that team’s play, they interchanged between messaging the support number and Traitel’s and gameplay was smooth and responsive the entire time. This was the only team of the entire event with such issues.

Around midday (3 hours after being contacted), Traitel contacted SCOOT support to inform that they had resolved the outgoing congestion issue. One of Telstra’s servers that services Traitel’s SMS system wasn’t working, thus the much higher rate of failure experienced earlier. Telstra quickly fixed their service, thus resuming Traitel’s service. By this time, MessageNet was being used heavily so the service was not switched back to Traitel.

At around 1pm another problem struck. MessageNet suddenly ceased to send out SMS. It seemed that a quota, previously unknown of, had been reached. Attempts were made by phone and email to contact MessageNet but calls were redirected to an answering service. An urgent message was left for MessageNet but there was never a reply. As a result, the outgoing SMS service was switched back to Traitel. Teams were still encouraged to SMS MessageNet over Traitel, just in case Traitel’s service became congested again. The 10-15 minute period of downtime threw several teams out of sync with the system but calls to support were sufficient to reorientate them.

This setup functioned effectively until the end of the day.

Day Three

Before the start of Day Three, it was decided to stick with MessageNet as the primary number (as their response times were quicker), but to raise the initially mysterious daily quota. A quick call to MessageNet support successfully raised the quota. It seemed the quota was set back when MessageNet was first integrated with SMS System (an outdated and publicly unlisted account type). The quota had never been reached before. The monthly plan was also modified so that SMS costs were lowered for the event. Teams from then on would always receive SMS from MessageNet. Traitel was kept as backup.

This setup functioned extremely well and no problems arose for the entire day.

Conclusion

MessageNet was a far stabler SMS service than Traitel. Though it still had issues with certain mobile phone plans, it had far quicker response times. Unfortunately neither company seemed to staff phone lines on the weekend but Traitel at least could be contacted through email. If similar issues arose with MessageNet as it did to Traitel, the event could’ve stalled for a far longer period of time.

The decision to have two SMS services ready to go proved critical to the event. Reaching MessageNet’s daily SMS limit on Day One would’ve been catastrophic as it wouldn’t have been resolved all weekend. Without an alternative, the event would’ve failed.

Initially settling with Traitel as the primary service also ended up as a fortunate decision. The primary issue of number congestion was sorted early in the event and Traitel became a somewhat reliable backup service to MessageNet. Unfortunately the congested dedicated number issue wasn’t and realistically couldn’t be tested until the actual event. Such consistent and heavy SMS traffic would have been costly to test. It is assumed that Traitel haven’t had such an experience before the event (or the system/equipment infrastructure to cope). To their credit, they managed to quickly arrive at a suitable solution.

MessageNet’s costs (17c / SMS) were much higher than Traitel’s (9.9c / SMS), but in the end it reflected in their far stabler system. If phone support was available 24/7 and issues with certain phone plans resolved, MessageNet would be the best solution in regards to stability. Cost-wise though is another story, especially since at one point there was talk of an upper limit of 50000 SMS sent for the entire event.

Hosting Issues

Administration of the SMS System during an event like SCOOT can be efficient and straightforward, but requires good knowledge of the SMSSystem and Third Party web interfaces. One could easily be trained to administer the SMS  System to provide very effective live support during an event.

The most important aspect of live support is enabled through several web interfaces. These interfaces were used to resolve the vast majority of gameplay related cases that arose during the running of the event.

  • phpMyAdmin allows an administrator to view all SMS data directly from the database. This was useful in determining the location of teams in the game path (one can use this knowledge to approximately physically locate and progress teams). SCOOT support staff could reorientate lost teams by knowing which SMS clues to resend. This was useful when teams stopped halfway through a path the first day and wished to continue the next day, or when critical clue SMS were not received due to service issues.
  • The Gameplay interface and Development and Administration Tools allowed support staff to send messages to the system as if from a team’s mobile phone. This was useful for reasons discussed previously but was also especially useful in one case where a team’s mobile phone plan was incompatible with third party SMS services (e.g. A specific Vodafone plan when used with the MessageNet service). As such, no team was excluded from playing in the event.
  • Third Party web interfaces were used to resend SMS to teams. Traitel’s web interface also indicated when an SMS was successfully received and when an SMS was successfully sent to a mobile number. This was very useful when diagnosing issues during play.

Game puzzle solution data could be modified live without affecting the smooth functioning of the event. This was useful when unanticipated puzzle solutions surfaced during the event. Live modification of game path data was also possible when it was required.

Support staff were able to switch to other SMS service providers live during the event. This was critical when issues arose with a particular SMS service, such as delayed SMS responses due to congestion/load or the rare event of sudden service downtime.

Other Issues

Full Message Inbox: When a team’s message inbox is full, they are unable to receive SMS from any system. SMS sent to a full inbox may also incur further delay due to retransmission methods. See below for further detail.

Black Spots: Some physical locations provide low or no reception for SMS communications. This is an issue when SMS clues are critical and require a quick response time. Teams that wander off the physical game path in order to attain better reception may get disorientated and require support. SMS sent to phones that are uncontactable may also incur further delay due to retransmission methods. See below for further detail.

Retransmissions: If an SMS cannot be received by a phone, further attempts are made but with each failed attempt the length of time between attempts increase. If a team enters an area with excellent reception, they may not receive a previously failed SMS until much later. Immediate retransmission of failed SMS can sometimes be triggered if a team sends a dummy SMS but the effectiveness of this method is not entirely confirmed.

C. Data Collection and Analysis

The SMS System stored the following user and gameplay data pertaining to the SCOOT event:

  • Team Details (username, mobile number)
  • Team Scores
  • Team Progress
  • Game Puzzles
  • Game Puzzle Path Sequence
  • Game Result Scoring and Custom SCOOT Details
  • All Sent and Received SMS

All other SCOOT related data is stored in the SCOOT Database (e.g. Virtual Game Scores, Team Avatars, Team Inventory).

Important data is easily extracted from the SCOOTdatabase. Most of the statistics listed in this report were gathered using simple SQL queries on several tables. Manual sorting and counting was required for several lots of statistics but such manual work could be automated with simple data parsing algorithms. The sorting and counting process could also be expedited by features in common text editing applications (Find and Replace in jEdit).

If data analysis becomes a more important and requested component of SCOOT (e.g. advertising data), tools could be easily created and the database optimised for such purposes.

A special thanks to Yang Wong who was the Lead Systems Developer for SCOOT and helped put this document together.