Add Edge Support to an ARGame

Build on top of a simple AR shooting game.

Version: 1.0
Last Modified: 6/25/2020

This workshop builds on top of a simple AR shooting game, which utilizes iOS’s ARKit library. We will introduce you to MobiledgeX's Distributed Matching Engine (DME) APIs and will be making use of a MobiledgeX's library:

  • The MobiledgeX MatchingEngine library exposes various services that MobiledgeX offers such as finding the nearest MobiledgeX Cloudlet Infrastructure for client-server communication or workload processing offload.

This library has been published to an Artifactory repository and we will be adding them to our workshop project. To interface with the library, we must update the workshop code to implement APIs. This document will walk you through how to do this.

Workshop Goals

  1. Starting with the Workshop Skeleton app, add Edge support to register the client, find the closest Edge cloudlet, and verify our device location.

  2. Gain understanding of Distributed Matching Engine APIs

  3. Gain understanding of suggested Developer workflow to connect to application backend

  4. Gain understanding of the Client-Server model using http://socket.io.

Prerequisites

  • Access to MobiledgeX Console and Artifactory

  • Get GitHub access to MobiledgeX, git, and a GitHub username

  • Create a Github SSH key on your machine, then add it here: https://github.com/settings/keys (passwordless Github usage via SSH keys)

  • MacOS Mojave installation

  • iOS simulator (all target iOS devices that support iOS 12.0 or higher)

  • Xcode 10 (App Store on MacOS, search for "Xcode" from Apple)

  • Apple ID: Main developer site: https://developer.apple.com, Click on Accounts (for the current website)

  • Cocoapods installation: https://cocoapods.org

Instructions

Get Project Source Code

First, we have to get the ARShooter Skeleton App code.

Locate where you want to have the ARShooter repo stored locally.

cd DIRPATH

OR create the directory that you want your edge-cloud-sampleapps repo to be located in:

mkdir <REPO>
cd <REPO>

Check out the edge-cloud-sampleapps repository from the the MobiledgeX Github:

git clone https://github.com/mobiledgex/edge-cloud-sampleapps.git

If you do not have git installed, follow the link https://www.atlassian.com/git/tutorials/install-git to install.

The following directory contains the project used in this article:

  • edge-cloud-sampleapps/iOS/ARShooterExample/ARShooterSkeleton/

 Import MatchingEngine Library

The MatchingEngine SDK is stored in Artifactory and we will pull it in using Cocoapods (an iOS dependency manager).

In terminal, run the following commands:

  • Install cocoapods
$ gem install cocoapods

(may have to use “sudo gem” if not a privileged user)

  • Install cocoapods-art
$ gem install cocoapods-art

(may have to use “sudo gem“ if not a privileged user)

  • Go to root directory
$ cd ~
  • Create a .netrc file and put in credentials:
$ echo "machine artifactory.mobiledgex.net login <ArtifactoryUsername> password <ArtifactoryPassword>" > .netrc

where <username> and <password> are your Artifactory/Console credentials.

$ pod repo-art add cocoapods-releases https://artifactory.mobiledgex.net/artifactory/api/pods/cocoapods-releases

These commands will pull in the Podspecs from the MobiledgeX Artifactory repo into your computer. You can see them by navigating to
~/.cocoapods/repos-art/cocoapods-releases/Specs/MobiledgeXiOSLibrary/2.1.0/MobiledgeXiOSLibrary.podspec

We will now go to the SkeletonApp. Navigate to edge-cloud-sampleapps/iOS/ARShooterExample/ARShooterSkeleton/ and then run:

pod install

This will look at the podspecs specified in the Podfile: source https://github.com/CocoaPods/Specs.git for Open Source dependencies we are using, and our local cocoapods-releases directory for the MobiledgeXiOSLibrary podspec. The podspec will inform Cocoapods where the source code is and CocoaPods will automatically integrate the dependencies into our project.

Implement MatchingEngine APIs

We will be implementing 3 MatchingEngine APIs:

RegisterClient() - Identifies the user (Organization Name) and application details (appName and appVersion), and allows the usage of MobiledgeX integration.

FindCloudlet() - Returns the edge application server to communicate with, in the form of an AppPort list that needs to be parsed to retrieve your particular application's server details. Either TCP or UDP transport. A cloudlet is a small-scale cloud datacenter.

VerifyLocation() - Returns the distance between the location that the user claims and what the carrier knows. Public fields such as getTowerStatus(), getGpsLocationStatus(), and getGpsLocationAccuracyKm() allow the developer to ensure the user is located where they claim to be.

The iOS MatchingEngine SDK uses the Promises framework to easily work with asynchronous code. We will be using the APIs to register the user, find the nearest edge server running the app (cloudlet), and verify the location of the user.

Open Xcode:

You will see an Xcode workspace in edge-cloud-sampleapps/iOS/ARShooterExample/ARShooterSkeleton/

Open Xcode and in the bottom right hand corner, click “Open another project…”

Navigate to edge-cloud-sampleapps/iOS/ARShooterExample/ARShooterSkeleton/ and double click on ARShooterSkeleton.xcworkspace to open the Xcode workspace.

Setup

First we have to setup MatchingEngine in our Client code and the variables that will be used in our APIs.

Go to LoginViewController/LoginViewController.swift:

First, underneath the other imports, copy and paste this:

import MobiledgeXiOSLibrary

Then, underneath this comment:

// MatchingEngine variables

Copy and paste the following:

var matchingEngine: MobiledgeXiOSLibrary.MatchingEngine!

Let us then declare important variables that will be used in our MatchingEngine API calls. In the function “setUpMatchingEngineConnection()”, replace:

SKToast.show(withMessage: "MatchingEngine not setup yet")

With:

matchingEngine = MobiledgeXiOSLibrary.MatchingEngine()
dmeHost = "wifi.dme.mobiledgex.net"
dmePort = 38001
appName = "ARShooter"
appVers = "1.0"
orgName = "MobiledgeX"
carrierName = ""
location = MobiledgeXiOSLibrary.MatchingEngine.Loc(latitude: 37.459609, longitude: -122.149349) // Get actual location and ask user for permission

RegisterClient:

Replace:

SKToast.show(withMessage: "RegisterClient not implemented yet")

With:

// Register user to begin using edge cloudlet
let registerClientRequest = matchingEngine.createRegisterClientRequest(
                                                orgName: orgName,
                                                appName: appName,
                                                appVers: appVers)

registerPromise = matchingEngine.registerClient(host: self.dmeHost!, port: self.dmePort!, request: registerClientRequest)
.then { registerClientReply in
    SKToast.show(withMessage: "RegisterClientReply: \(registerClientReply)")
    print("RegisterClientReply: \(registerClientReply)")

    SKToast.show(withMessage: "FindCloudlet not implemented yet")

    SKToast.show(withMessage: "VerifyLocation not implemented yet")
}

Because we need the Session Cookie that is returned by RegisterClientReply before we can use the other APIs, we use a closure to make sure FindCloudlet and VerifyLocation APIs are only called once RegisterClientReply returns.

However, FindCloudlet and VerifyLocation do not depend on each other and so we can run them together asynchronously.

FindCloudlet:

Replace:

SKToast.show(withMessage: "FindCloudlet not implemented yet")

With:

// Find closest edge cloudlet
let findCloudletRequest = self.matchingEngine.createFindCloudletRequest(gpsLocation: self.location!, carrierName: self.carrierName!)
self.findCloudletPromise = self.matchingEngine.findCloudlet(host: self.dmeHost!, port: self.dmePort!, request: findCloudletRequest)


all(self.findCloudletPromise!)
.then { value in
    // Handle findCloudlet reply
    let findCloudletReply = value.0
    SKToast.show(withMessage: "FindCloudletReply is \(findCloudletReply)")
    print("FindCloudletReply is \(findCloudletReply)")

    // Handle verifyLocation reply
    SKToast.show(withMessage: "Need to handle verifyLocation reply")
}.catch { error in
    // Handle Errors
    SKToast.show(withMessage: "Error occured in callMatchingEngineAPIs. Error is \(error.localizedDescription")
    print("Error occured in callMatchingEngineAPIs. Error is \(error.localizedDescription)")
}

VerifyLocation:

Delete:

SKToast.show(withMessage: "VerifyLocation not implemented yet")

Change this line: all(self.findCloudletPromise!) to:

all(self.findCloudletPromise!, self.verifyLocationPromise!)

Then copy and paste the following code right ABOVE it:

let verifyLocationRequest = self.matchingEngine.createVerifyLocationRequest(gpsLocation: self.location!, carrierName: self.carrierName!)
self.verifyLocationPromise = self.matchingEngine.verifyLocation(host: self.dmeHost!, port: self.dmePort!, request: verifyLocationRequest)


Replace:

SKToast.show(withMessage: "Need to handle verifyLocation reply")

With:

let verifyLocationReply = value.1
SKToast.show(withMessage: "VerifyLocationReply is \(verifyLocationReply)")
print("VerifyLocationReply is \(verifyLocationReply)")

GetConnection workflow:

The GetConnection workflow is the suggested workflow to register the user using an application, find the nearest cloudlet with the application backend deployed, and get a “connection” object to send and receive data.

The workflow is:

  1. RegisterAndFindCloudlet(): Wrapper function for registerClient() and findCloudlet(). Returns a dictionary with findCloudletReply fields.

  2. Get[Protocol]AppPorts(): Returns a dictionary (key: String, value: [String: Any]), where keys are the internal port specified on app deployment and values are the AppPort “object” returned in the ports field of findCloudletReply. (This object may contain a range of ports and an fqdn prefix that is specific to the application backend)

    1. The developer will know their internal port and will get the AppPort “object” from the dictionary that corresponds to that internal port. The developer will use this in the GetConnection() call.
  3. Get[Protocol]Connection(): Depending on the protocol (TCP, UDP, HTTP, Websockets), this will return a different Swift object to be used to send and receive data.

Replace:

SKToast.show(withMessage: "GetConnection workflow not implemented yet")

With:

let replyPromise = matchingEngine.registerAndFindCloudlet(
                                                          orgName: orgName,
                                                          gpsLocation: location!,
                                                          appName: appName,
                                                          appVers: appVers,
                                                          carrierName: carrierName)
.then { findCloudletReply -> Promise<SocketManager> in
    // Get Dictionary: key -> internal port, value -> AppPort Dictionary
    guard let appPortsDict = try self.matchingEngine.getTCPAppPorts(findCloudletReply: findCloudletReply) else {
        throw LoginViewControllerError.runtimeError("GetTCPPorts returned nil")
    }
    if appPortsDict.capacity == 0 {
        throw LoginViewControllerError.runtimeError("No AppPorts in dictionary")
    }
    // Select AppPort Dictionary corresponding to internal port 3001
    guard let appPort = appPortsDict[self.internalPort] else {
        throw LoginViewControllerError.runtimeError("No app ports with specified internal port")
    }

    return self.matchingEngine.getWebsocketConnection(findCloudletReply: findCloudletReply, appPort: appPort, desiredPort: Int(self.internalPort), timeout: 5000)

}.then { manager in
    self.manager = manager
}.catch { error in
    print("Error is \(error)")
}

Connect to an Edge Server for Multiplayer:

After implementing these APIs, we will know the fqdn (Full Qualified Domain Name) of the nearest Cloudlet running the server of our app. Note: FindCloudlet returns the hostname and port that we must use to connect to that server.

This AR app requires a gameID and a username to be inputted before turning on the camera.

Replace:

SKToast.show(withMessage: "Pass MatchingEngine and GameState variables to GameViewController")

With:

// Set variables for next GameViewController
gameViewController.gameID = gameID
gameViewController.userName = userName
gameViewController.peers[userName!] = 0
gameViewController.manager = self.manager

We must connect the nearest backend server (which the SocketManager is configured to connect to) to join a multiplayer game and send data between different clients running the game. We then pass these variables to the gameViewController which will handle the actual AR game.

Take a look at the GameViewController folder for the code that implements the ARShooter game. It utilizes the ARKit library to create and render AR objects through your phone’s camera. GameViewController.swift holds the main gameplay logic, and the other files are Delegate and Callback files to handle the results of different ARKit functions.

Notice in GameViewController, the lines of code “socket.emit…“. These calls send data to the server to handle. In SocketIOManager.swift, we implement the callbacks for when the device receives data from the server.

Server Code:

Originally, this app used iOS’s MultipeerConnectivity Framework, which uses peer to peer communication. This framework breaks down with more than 7 or 8 players connected to the same game. So, we have to use a server to handle communication between devices and synchronize the game state (so everyone sees the same thing, or at least close to the same thing).

To see the server code, go to edge-cloud-sampleapps/iOS/ARShooterExample/ARShooterServer/shooter-server.js. This a bare-bones Node.js server that directs communication between devices and can keep track of different games. It uses the socket.io library, which is mirrored in client side with the Socket.IO-Client-Swift library (which we imported using Cocoapods).

For testing purposes, you can run the server locally, and change the Client socket to match your local ip address and in edge-cloud-sampleapps/iOS/ARShooterExample/ARShooterServer/, run “node shooter-server.js”.

As stated at the beginning, look at this article: Converting P2P App to Client Server for more info on the structure of server side socket.io code.

User Location

The MobiledgeXiOSLibrary requires user location to find the nearest cloudlet.

The previous code hardcoded the user location for ease of use. An actual application would request location permissions from the user and grab the gps location whenever needed. This section will show how to request location permissions and convert GPS locations to a MobiledgeX.Loc object that can be used in FindCloudlet and VerifyLocation calls.

Request Permissions

First let us request location permissions from the user. Find and replace:

SKToast.show(withMessage: "Request location permissions not implemented yet")

With:

self.locationManager = CLLocationManager()
locationManager!.delegate = self
locationManager!.requestWhenInUseAuthorization()

Note, LoginViewController implements the CLLocationManagerDelegate interface. We will implement a delegate function in this file, so we must assign LoginViewController to be the locationManager's delegate.

Next, let's delete the call that initiates our MatchingEngine and GetWebsocketConnection functions, so that we do not start calling MatchingEngine APIs before we have the user's location:
In the viewDidLoad() function, delete:

mobiledgeXIntegration()

We can also delete the code that initializes the location variable. In setUpMatchingEngineParameters() function, delete:

location = MobiledgeXiOSLibrary.MatchingEngine.Loc(latitude: 37.459609, longitude: -122.149349) // Get actual location and ask user for permission

Implement CLLocationManagerDelegate

Finally, let's implement our CLLocationManagerDelegate function. This function will be called when the app starts and whenever the user changes location permissions (ie. after they allow location permissions from previous code).

Once the user allows location permissions, we will grab the user's gps location and then convert it to a MobiledgeX.Loc object to be used in FindCloudlet and VerifyLocation.

Replace:

SKToast.show(withMessage: "Location Manager delegate not implemented yet")

With:

var currLocation: CLLocation!

if (status == .authorizedAlways || status == .authorizedWhenInUse) {
    currLocation = manager.location
    location = MobiledgeXiOSLibrary.MatchingEngine.Loc(latitude: currLocation.coordinate.latitude, longitude: currLocation.coordinate.longitude)
}

if location != nil {
    mobiledgeXIntegration()
}

Hackathon Tasks

  1. Make the AR game better

    • Handle collisions between AR objects and real life objects
    • Make it a laser tag game → If you hit another person’s phone with the AR bullet, they lose the game
    • Improve the stability of the “origin” and the worldmap (objects don’t always stay in the same place as you move)
    • Change physics to make it cooler
    • Make the game look nicer
  2. Improve the server

    • Add game synchronization → server handles scoring, saves world map point cloud, improves on point cloud if more users are in the game
  3. Improve “WorldMap”

    • Correlate cloud maps and determine if two users are in the same room
    • Combine point clouds, so that users don’t have to walk all around the room to get an accurate point cloud.