How do we test embedded wireless systems?
Let's take a fictitious product, a Remote Pilot System to see the process.
Our imaginary Remote Pilot System (RPS)
lets pilots guide an aircraft from outside the cockpit, say from the First
Class cabin, or even from the ground. The cockpit transmits all data to the
RPS. The pilot studies the RPS console and instructs the controls in the
cockpit. He might even be miles away from the cockpit.
For now, let's assume that the pilot is
seated in the First Class cabin. The RPS connects to the cockpit via a 802.11x
wireless protocol. The RPS talks to the dials and joysticks in the cockpit
using a custom protocol, layered atop the standard wireless protocol.
That's enough background. On to the
First, figure out the threats to the
system. That's the Threat Profile in our lingo.
... hijacks the controls of the plane from
... tampers with the instructions sent by
... messes the dial readings the pilot sees
... knocks out the connection between the
RPS and the cockpit
Probably not exhaustive, but those are the
most important ones. Good enough. Let's move on.
Next, we create the Test Plan. The
Test Plan maps each threat to different lines of attack. Examples of
"lines of attack" are: replay attacks, sniffing, jamming, etc.
Let's take one threat from our Threat
Profile and figure out a test plan. Consider: "An adversary tampers with
the instructions sent by the pilot". Here're different lines of attack we
an instruction sent by the pilot: when the pilot instructs the plane to climb
5000 feet, capture the request, and replay it multiple times to send the plane
the request via a man-in-the-middle attack and modify the variables: change
that 5000 feet to 25000 feet and see what happens.
instructions the pilot never made: unlock the doors, turn off radio contact,
reduce the speed
We think through and prepare the test plan.
The better the test plan, the stronger the test.
Once the Test Plan is ready, figure out
the exact Test Cases for each line of attack. Let's take an example:
Replay an instruction sent by the pilot.
How does the test case for this look? Remember the RPS is talking over a
Wireless LAN and it's no doubt using some form of encryption. So, step #1 is to
identify the encryption scheme. Next capture enough packets to try and break
it. If we break it, we then locate the packets with instruction from the pilot.
Find out an instruction that's sent in just one packet, say "Climb 5000
feet now!" We are really figuring out the RPS custom protocol here.
Recreate wireless frames for that
instruction, spoofing it to come from the pilot and re-inject it back in the
air. Watch our balance to see if the test succeeded.
Ok, I was trying to be funny. But that's
the basic approach for testing an embedded wireless system: Threat Profile to Test Plan to Test Cases
and Cons of Black Box Testing
Here're the pluses and minuses of a black
box test for wireless embedded systems.
First the minus: it's not very efficient.
A black box pen tester spends a fair bit of
time trying to break the encryption, interpreting the packets, constructing
test cases. An in-depth review of architecture and protocols can be more
thorough in spotting weaknesses in lesser time.
And the plus: it’s more realistic in two
One, a black box test tells you how hard it
is to actually break in. No architecture review can tell you if a hole can be
compromised in a week or in a month. The penetration test simulates the
travails of the outside attacker best.
Two, the architecture and protocol rules
are finally documents. The implementation, in practice, might vary a bit from
the original design, and that might introduce holes. Testing a deployed system
gives you greater assurance on the implementation.
A combination of black box testing with
design review provides the best value. The black box test alone might not find
all holes in the limited time for a test. A review of the architecture and
protocol done after the black box test can pick out flaws faster.