This document describes how to execute automated LAVA tests controlling resources
external to the DUT across a network implementing a LAVA parallel pipeline job.
Test Cases
The approach proposed in this document will help to address test cases like:
-
Executing a test in the DUT where certain power states are simulated (for
example a power loss) during specific test actions using a programmable PSU
external to the DUT.
-
Executing a test in the DUT simulating SD card insertion and removal using
an external device.
The only assumption, in both scenario, proposed in this document is that the
external device (either a programmable PSU or SD-card simulator) can be accessed
through the network using SSH.
LAVA Features
LAVA offers the following features that can be combined to implement a solution
for the test cases mentioned in this document:
- LXC to deploy required software and tools to access the external device.
- MultiNode to communicate data between jobs actions.
- Secondary connections for executing tests through SSH.
LXC
LAVA supports LXC containers both as a standalone device type and as dynamic
transparent environments in order to interact with external devices. In either
case the LXC Protocol is used.
MultiNode
The MultiNode Protocol allows data to be shared between actions, including data
generated in one test shell definition being made available over the protocol to
a deploy or boot action of jobs with a different role.
Synchronisation is done using the MultiNode API, specifically the lava-send
and
lava-wait
calls.
Secondary Connections
LAVA allows Secondary Connections to open network connections to external
devices using MultiNode submissions.
Approach Overview
The main idea is to create a LXC container device associated to the DUT responsible
to execute the automated test, then opens a SSH connection to an external device,
and use the MultiNode API in order to synchronize both devices and pass data
between them with the LXC container serving like a coordinator of the different
LAVA tests actions.
In this way, a server-client layout is setup that will help to execute tests in a
board attached to LAVA (server side) with intervention of external devices (client
side).
LAVA Job Connection Layout
The LXC container is deployed directly from the LAVA dispatcher and coordinate
the execution of the parallel pipeline between the DUT and the external device
(secondary connection) from there.
The layout model would be something like:
------------- DUT
/ MultiNode
LAVA (LXC)
\
------------- Secondary Connection (PSU, SD-Card HW)
MultiNode
Test Job
This section shows the basics proposed in this document using a LAVA job file
example.
The following steps describe the main flow of the job:
1 - Create two types of roles host
and guest
. The host
role will contain the
LXC container and the DUT, the guest
role will label the SSH connection for
the external device. This creates two groups (host
and guest
) that can
communicate using the MultiNode API, so messages can be sent between the LXC
and Device as the server and the secondary connection as the client.
2 - Label both types of roles in the protocols
section of the job.
3 - Deploy and boot the LXC
container (host
).
4 - Execute a test in the LXC container using the MultiNode API to send the
lava_start
message, so the deploy
action for the external device can start,
and waits for remaining clients to start using the lava-sync
call.
5 - Deploy the DUT (host
).
6 - Deploy the external device (guest
) , which is waiting for the LXC
lava_start
message to start deployment. Once this message is recevied, the
guest device is deployed.
7 - Boot DUT.
8 - Boot external device.
9 - Execute a test in the DUT sending the lava-sync
call.
10 - Execute a test in the external device sending the lava-sync
call.
11 - Once all clients are synchronized (the LXC, DUT and external device), start
executing tests.
12 - Tests executed in the DUT and external device needs to use the MultiNodeAPI
in order to pass data between them.
As the LXC is deployed and booted first, the LXC can run a test shell before
deploying the device, before booting the device, before the test shell action on
the device which starts the secondary connection guests or at any later point
(AddingTestsActions).
Job File Example
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
|
job_name: LXC and Secondary connection with a Device
timeouts:
job:
minutes: 30
action:
minutes: 3
connection:
minutes: 5
priority: medium
visibility: public
protocols:
lava-lxc:
host:
name: lxc-ssh-test
template: debian
distribution: debian
release: stretch
lava-multinode:
# expect_role is used by the dispatcher and is part of delay_start
# host_role is used by the scheduler, unrelated to delay_start.
roles:
host:
device_type: beaglebone-black
# This makes this role essential in order to execute the test.
essential: True
count: 1
timeout:
minutes: 10
guest:
# protocol API call to make during protocol setup
request: lava-start
# set the role for which this role will wait
expect_role: host
timeout:
minutes: 15
# no device_type, just a connection
connection: ssh
count: 3
# each ssh connection will attempt to connect to the device of role 'host'
host_role: host
actions:
- deploy:
role:
- host
namespace: probe
timeout:
minutes: 5
to: lxc
# authorize for ssh adds the ssh public key to authorized_keys
authorize: ssh
packages:
- usbutils
- procps
- lsb-release
- util-linux
- ntpdate
- openssh-server
- net-tools
- boot:
role:
- host
namespace: probe
prompts:
- 'root@(.*):/#'
timeout:
minutes: 5
method: lxc
- test:
role:
- host
namespace: probe
timeout:
minutes: 5
definitions:
- repository:
metadata:
format: Lava-Test Test Definition 1.0
name: network
description: "Send message ID"
run:
steps:
- lava-test-case ntpdate --shell ntpdate-debian
- lava-echo-ipv4 eth0
- lava-send ipv4 ipaddr=$(lava-echo-ipv4 eth0)
- lava-send lava_start
- lava-sync clients
from: inline
name: lxc-test
path: inline/lxc-test.yaml
# DUT actions
- deploy:
role:
- host
namespace: device
timeout:
minutes: 5
to: tftp
kernel:
url: https://files.lavasoftware.org/components/lava/standard/debian/stretch/armhf/3/vmlinuz-4.9.0-4-armmp
sha256sum: b6043cc5a07e2cead3f7f098018e7706ea7840eece2a456ba5fcfaddaf98a21e
type: zimage
ramdisk:
url: https://files.lavasoftware.org/components/lava/standard/debian/stretch/armhf/3/initrd.img-4.9.0-4-armmp
sha256sum: 4cc25f499ae74e72b5d74c9c5e65e143de8c2e3b019f5d1781abbf519479b843
compression: gz
modules:
url: https://files.lavasoftware.org/components/lava/standard/debian/stretch/armhf/3/modules.tar.gz
sha256sum: 10e6930e9282dd44905cfd3f3a2d5a5058a1d400374afb2619412554e1067d58
compression: gz
nfsrootfs:
url: https://files.lavasoftware.org/components/lava/standard/debian/stretch/armhf/3/stretch-armhf-nfs.tar.gz
sha256sum: 46d18f339ac973359e8ac507e5258b620709add94cf5e09a858d936ace38f698
compression: gz
dtb:
url: https://files.lavasoftware.org/components/lava/standard/debian/stretch/armhf/3/dtbs/am335x-boneblack.dtb
sha256sum: c4c461712bf52af7d020e78678e20fc946f1d9b9552ef26fd07ae85c5373ece9
- deploy:
role:
- guest
namespace: guest
# Timeout for the ssh connection attempt
timeout:
seconds: 30
to: ssh
connection: ssh
protocols:
lava-multinode:
- action: prepare-scp-overlay
request: lava-wait
messageID: ipv4
message:
ipaddr: $ipaddr
timeout: # delay_start timeout
minutes: 5
- boot:
role:
- host
namespace: device
timeout:
minutes: 15
method: u-boot
commands: nfs
auto_login:
login_prompt: 'login:'
username: root
prompts:
- 'root@stretch:'
parameters:
shutdown-message: "reboot: Restarting system"
- boot:
role:
- guest
namespace: guest
timeout:
minutes: 3
prompts:
- 'root@stretch:'
parameters:
hostID: ipv4
host_key: ipaddr
method: ssh
connection: ssh
- test:
role:
- host
namespace: device
timeout:
minutes: 30
definitions:
- repository:
metadata:
format: Lava-Test Test Definition 1.0
name: install-ssh
description: "install step"
run:
steps:
- df -h
- free
- lava-sync clients
from: inline
name: ssh-inline
path: inline/ssh-install.yaml
- repository: http://git.linaro.org/lava-team/lava-functional-tests.git
from: git
path: lava-test-shell/smoke-tests-basic.yaml
name: smoke-tests
- repository: http://git.linaro.org/lava-team/lava-functional-tests.git
from: git
path: lava-test-shell/single-node/singlenode02.yaml
name: singlenode-intermediate
- test:
role:
- guest
namespace: guest
timeout:
minutes: 5
definitions:
- repository: http://git.linaro.org/lava-team/lava-functional-tests.git
from: git
path: lava-test-shell/smoke-tests-basic.yaml
name: smoke-tests
# run the inline last as the host is waiting for this final sync.
- repository:
metadata:
format: Lava-Test Test Definition 1.0
name: client-ssh
description: "client complete"
run:
steps:
- df -h
- free
- lava-sync clients
from: inline
name: ssh-client
path: inline/ssh-client.yaml
#
# Tests executed in the external device and DUT can be added here.
# They all need to use the MultiNode API.
#
# Execute test in the DUT
- test:
role:
- host
namespace: device
timeout:
minutes: 10
definitions:
- repository: https://gitlab.apertis.org/tests/apertis-test-cases/
from: git
path: lava-test-shell/single-node/singlenode03.yaml
name: singlenode-advanced
# Execute test in the external device (PSU, SD-card device)
- test:
role:
- guest
namespace: guest
timeout:
minutes: 10
definitions:
- repository: https://gitlab.apertis.org/tests/apertis-test-cases/
from: git
path: lava-test-shell/single-node/singlenode03.yaml
name: singlenode-advanced
|
QA Report
Once tests results are available at LAVA , and the test cases are enabled for the
specific images from the test case repository, the results will be available from
the QA Report App automatically.