← back to the blog


LC3 Automation Tutorial

Posted in New LC3 Simulator

Lets walk through a simple LC3 test automation script. The python unittest library provides a simple, yet powerful, way of automating experiments.

 

Prompt

In this assignment, students are asked to sort in-place a list of (student_id, grade) tuples. The program starts at address x3000, the list at x3200, and the list must be sorted by grade in descending order. A list entry is a 16 bit word where the upper half, bits[15:8], and the lower half, bits[7:0], correspond to a unique student id and grade respectively. 

  • Program starts at x3000
  • List starts at x3200
  • list entry 
    • bits[15:8] = student_id (unique)
    • bits[7:0]   = grade
  • Sorted in descending order by grade
  • Sorted in-place

Let's start by creating a new test script, sortingTest.py


#!/usr/bin/env python2.7

import pylc3
import unittest

# Some Helper Functions

# Our simulator only accepts unsigned 16 bit numbers
# This function just forces python to treat int32 as uint16 :) 
def neg(x):
  return x & 0xFFFF

def upperHalf(x):
  return (x << 8) & 0xFF00

def lowerHalf(x):
  return x & 0x00FF

Note we build against Python 2.7, at the moment Python 3 does not work.

 

Creating A Test Harness

The python unittest library provides a convenient way of creating, discovering, managing, and running test cases; all we have to do is extend the unittest.TestCase class in our derived class.  The TestCase class provides a setUp routine which gets called before each test. We can override this routine to reduce redundant code, such as creating a pyLC3 simulator instance or loading an LC3 object file. The actual tests are fairly simple to construct. According to the documentation, a TestCase test is just a python member function prepended by test_. To tell whether a test passes or fails, we usually include one or more assertions (ex assertEqual(x,y)  means test passes if and only if x == y).  

Here is an example test case:

 


class myFirstTestCase(unittest.TestCase):

  #The Setup class is provided by unittest and is run before each test
  def setUp(self):
    self.delicious = True

  def test_byteMe(self):
    x = 5
    ... Do something with x
    self.assertEqual(x,5)

  def test_byte_cookie(self):
    cookie = True
    self.assertEqual(cookie, delicious)
  
  def test_byteMe_less_than(self):
    x = 5
    self.assertLT(x, 6) # Test Passes only if x < 6
    

This test case contains a simple setUp and 3 tests to run. We provide a simple way of running test cases and displaying the results in grader.py. This can be run with the following line in our script:


grader.doTest(myFirstTestCase)

 

Creating An Assignment Test Case

 Using what we learned in the previous section, we can define a simple SortingTest class:

 


class SortingTest(unittest.TestCase):

  #The Setup class is provided by unittest and is run before each test
  def setUp(self):
    self.sim = pylc3.simulator()
    self.sim.load("sortscores.obj")

 

Like before, we start by creating a class derived from TestCase. Notice in setUp routine we create a new simulator object? The simulator object contains the simulator in addition to access functions.

Here is a brief list of some of the main access functions:

  • stepN: Step N instructions
  • run: Run the simulator until halt reached
  • mem: Access simulator memory (python style)
  • setReg
  • getReg
  • addWatchPoint
  • load: Load an objectfile
  • getPC
  • setPC
  • + More to come :)

Something really cool about the simulator, we can access memory using python style symantics. This means we can use the slicing operator! Note, however, memory does not return a list directly. Though it is possible to cast the object to a list.

 


sim = pylc3.simulator()
x = sim.mem[range(10)] # Get the first ten elements from mem
print list(x) # Cast x to a list for easy printing
x[5] = 50
sim.mem[range(10)] = x
x = range(10,20)
sim.mem[range(10,20)] = x #Writing with lists also works
sim.mem[x3000] = 0xBEEF

As you can see, there are lots of ways we can populate memory. 

 

Adding the First Test

In our first example, let's assume the assignment prompt provides an example list to sort:

  1. student: 23 Grade: 78
  2. student: 10 Grade: 91
  3. student: 56 Grade: 5
  4. student: 2   Grade: 78

Let's add this test to our test case:

 


  def test_ProvidedInDocument(self):
    gradeList = [upperHalf(23) | lowerHalf(78), \
                 upperHalf(10) | lowerHalf(91), \
                 upperHalf(56) | lowerHalf(5),  \
                 upperHalf(2) | lowerHalf(87)]
    
    self.sim.mem[0x3200:(0x3200 + len(gradeList))] = gradeList
    self.sim.run()

    progOutList = self.sim.mem[0x3200:(0x3200 + len(gradeList))]

    #Python Sorts are guaranteed to be stable, so we can check if their output is correctly sorted
    # in just one line :) (We dont care whether the student implemented stable vs unstable sort)
    # Note, mem returns a pylc3.mem object, so we need to cast it to a list to print
    Expected = progOutList # Sorted Sorts the list in place
    sorted(Expected, key = lambda grade : grade & 0xFF, reverse=True)
    self.assertEqual(Expected, progOutList)

The lambda keyword in the sorted function is super cool. Lambdas allow us to write temporary functions without having to go through the standard def FOO(x): syntax. In this case we want to sort our list where the keys correspond to the students grade. The reverse=True keyword just means we are sorting in descending order. 

 

Creating a Randomly Generated Test

Being the clever students we are, we decide to write an additional test with randomly generated inputs. And since we are so ever so clever, we decide to copy and paste the code from the previous section making minor changes in the process. 

 


  # Generate A random Test
  def test_random_1(self):
    numStudents = 50
    gradeList = []  #Declare a list
    
    #Create the Grade list
    for i in range(numStudents):
      gradeList.append( upperHalf(i + 1) | lowerHalf(random.randint(0,100)))

    #Populate the simulator memory and run
    self.sim.mem[0x3200:(0x3200 + len(gradeList))] = gradeList
    self.sim.run()

    progOutList = self.sim.mem[0x3200:(0x3200 + len(gradeList))]

    #Python Sorts are guaranteed to be stable, so we can check if their output is correctly sorted
    # in just one line :) (We dont care whether the student implemented stable vs unstable sort)
    Expected = progOutList  # Sorted Sorts the list in place
    sorted(Expected, key = lambda grade : grade & 0xFF, reverse=True)
    self.assertEqual(Expected, progOutList)

Note we needed to add import random to our imports list to get this to work.

 

The Grader

Lastly, we need a grader to handle the tests. Here is a simple grader that prints out the tests that pass vs fail. Create a new file called grader.py


#!/usr/bin/env python2.7

import unittest

def doTest(testCase):

  suite = unittest.TestLoader().loadTestsFromTestCase(testCase)
  res = unittest.TextTestRunner(verbosity=2).run(suite)

def doTestVerbose(testCase):
  print "-------------------------------------"

  suite = unittest.TestLoader().loadTestsFromTestCase(testCase)
  res = unittest.TextTestRunner(verbosity=2).run(suite)

  print "List of Test Errors", res.errors
  print "List of Test Failures", res.failures
  print "List of Test Unexpected Successes", res.unexpectedSuccesses
  print "Num Tests Run:", res.testsRun
  print "Was Successful ?:", res.wasSuccessful()
  print "-------------------------------------"

The Final Code

Here is the completed code:

 


#!/usr/bin/env python2.7
"""Sorting Simulation
    This script is meant to simulate testing + grading of a LC3 programming Assignment
    Assignment: sort a list of (student_id, grades) in place. 
      Note: sorted by grade in descending order
    
    List Entries:
      student_id: uint8_t (unique)
      grade: uint8_t in [0,100]

    Student Program Starts at x3000
    Grade List starts at x3200
"""
import pylc3
import unittest
import random
import grader

# Helper Functions
def upperHalf(x):
  return (x << 8) & 0xFF00

def lowerHalf(x):
  return x & 0xFF

#not used in this script but necessary if writing negative numbers
def neg(x):
  return x & 0xFFFF

class SortingTest(unittest.TestCase):

  #The Setup class is provided by unittest and is run before each test
  def setUp(self):
    self.sim = pylc3.simulator()
    self.sim.load("sortscores.obj")
    # print("Done Initializing Simulator")

  def test_ProvidedInDocument(self):
    gradeList = [upperHalf(23) | lowerHalf(78), \
                 upperHalf(10) | lowerHalf(91), \
                 upperHalf(56) | lowerHalf(5),  \
                 upperHalf(2) | lowerHalf(87)]
    
    self.sim.mem[0x3200:(0x3200 + len(gradeList))] = gradeList
    self.sim.run()

    progOutList = self.sim.mem[0x3200:(0x3200 + len(gradeList))]

    #Python Sorts are guaranteed to be stable, so we can check if their output is correctly sorted
    # in just one line :) (We dont care whether the student implemented stable vs unstable sort)
    # Note, mem returns a pylc3.mem object, so we need to cast it to a list to print
    Expected = progOutList # Sorted Sorts the list in place
    sorted(Expected, key = lambda grade : grade & 0xFF, reverse=True)
    self.assertEqual(Expected, progOutList)

  # Generate A random Test
  def test_random_1(self):
    numStudents = 50
    gradeList = []  #Declare a list
    
    #Create the Grade list
    for i in range(numStudents):
      gradeList.append( upperHalf(i + 1) | lowerHalf(random.randint(0,100)))

    #Populate the simulator memory and run
    self.sim.mem[0x3200:(0x3200 + len(gradeList))] = gradeList
    self.sim.run()

    progOutList = self.sim.mem[0x3200:(0x3200 + len(gradeList))]

    #Python Sorts are guaranteed to be stable, so we can check if their output is correctly sorted
    # in just one line :) (We dont care whether the student implemented stable vs unstable sort)
    Expected = progOutList  # Sorted Sorts the list in place
    sorted(Expected, key = lambda grade : grade & 0xFF, reverse=True)
    self.assertEqual(Expected, progOutList)


grader.doTest(SortingTest)

Directory Structure

Here is what is in the current directory. Note, __init__.py is a empty file. This file tells python to treat the current directory as a package.

  • .
  • ..
  • sortingTest.py
  • sortscores.obj
  • grader.py
  • __init__.py