Legal Entity/Division - Date Multicore For Avionics Certification Issue 2013 – 03 – 22

Lega

l Ent

ity/D

ivis

ion

- D

ate

Multicore For AvionicsCertification Issue

2013 – 03 – 22

2 /2 /

Ce document est la propriété de Thales Group et il ne peut être reproduit ou communiqué sans autorisation écrite de Thales S.A.This document is the property of Thales Group and may not be copied or communicated without written consent of Thales S.A.

Context 1/2

This presentation is based on the final report that concludes the MULCORS project contracted with EASA.

The reports provides the main outputs, recommendations and conclusions per EASA Specifications attached to the Invitation to Tender EASA.2011.OP.30.

Access to MULCORS report https://www.easa.europa.eu/safety-and-research/researc

h-projects/large-aeroplanes.php

https://www.easa.europa.eu/safety-and-research/research-projects/large-aeroplanes.php

https://www.easa.europa.eu/safety-and-research/research-projects/large-aeroplanes.php

3 /3 /


Context 2/2

CONTEXT

Provide a survey of Multi-core processors market availability Define multi-core processors assessment & selection criteria Perform investigations on a representative multi-core processor Identify mitigation means, design and usage rules & limitations Suggest recommendations for multi-core processor introduction Suggest complementary or modification to EASA guidance

BACKGROUND

Digital Embedded Aircraft Systems Use of COTS processors in Embedded Aircraft Equipment Use of Multi-Core in Embedded Military Aircraft Equipment

4 /4 /


AGENDA

Multi-core: Introduction Problems to Solve Regarding certification Software Aspects Failure Mitigation Means & COTS Relative

Features

Conclusion

5 /5 /


MULTI-COREIntroduction

6 /6 /


Multi-Core: Introduction

Multi-Core processor Architecture: Unified Memory Access

Multi-core processors architecture is organized around one memory shared between all cores

Architecture requiring arbitration management on one hand and integrity mechanisms on the other hand to manage communication between cores and synchronization if required

In multi-core processors we need to take care about how Cache Memory Coherency is assumed

7 /7 /



Multi-Core processor Architecture: Distributed Architecture

Each core has the use of a dedicated memory with or without dedicated cache depending on the processor architecture

Memory Cache Management is simplified and occurs in the same way as in a single core processor (separate cache and memory are dedicated to each core).

8 /8 /



Multi-Core processor Architecture: Single Address space, Distributed Memory

Cores have their own cache, they can also have dedicated memory but they can have access to other core memories using the bus or the Network

In some multi-core architecture, the cluster bus is also part of the global network. In this variant of architecture, the bandwidth is at least dimensioned to sustain all the transfers in a cluster without causing perturbation to the others

9 /9 /



EXT MEMORY

Core

Cache

BUS

INTERCONNECT

Register Register RegisterRegister

Register Register

Core

Cache

Core

Cache

Core

Cache

BUSRegister Register

Core

Cache

Core

Cache

EXT MEMORY

Ex

tern

al

Bu

s

Ex

tern

al

Ne

two

rk

BSP BSP BSP

Hypervisor

O.S. O.S. O.S.

Drivers Drivers Drivers

Airb. SW Airb. SW Airb. SW Intended Function

HW adaptation Layer (BSP) Hypervisor layer (when required) Operating System Drivers Airborne Software

10 /10 /


MULTI-COREProblems to Solve

11 /11 /



What is a multicore processor? A multicore processor can be characterized by N (N ≥ 2) processing cores + a

set of shared resources (Memories, PCIe, Ethernet, Cache, Registers, etc.)

Two types of processors can be found The ones where interconnect between cores is based on an arbitrated bus

The ones where interconnect between cores is based on a network

Multicore management can be summarize to shared resources conflicts management (when SW is in DAL_A, DAL_B or DAL_C)

12 /12 /


Access conflits

To interconnect between cores If InterConnect = bus Access arbitration is done at this level

If InterConnect = network Access arbitration depend of numbers of authorized parallel routes (example : Memories accesses, Bus accesses, Networks accesses, etc.)


Si InterConnect = BUSSi InterConnect = Réseau

ConflictsManagement

ConflictsManagement

ConflictsManagement

ConflictsManagement

ConflictsManagement

13 /13 /



Accesses conflicts

To external Memories If InterConnect = bus Accesses arbitration has been realized at InterConnect level

If InterConnect = network Accesses arbitration are done at Memory Controller level In case of more than one Memory Controller, arbitration can be simplified

Si InterConnect = BUSSi InterConnect = RéseauGestion

des conflits

Gestion des

conflits

Si InterConnect = Réseau

Gestion des

conflits

Gestion des

conflits

14 /14 /



Accesses conflicts

Accesses to PCI / PCIe bus or ETHERNET Network If InterConnect = bus Accesses arbitration has been realized at InterConnect level

If InterConnect = network Accesses arbitration is done at each controller level: PCI / PCIe bus one or Ethernet network one

Depending of numbers of Accesses Controller, arbitration can be simplified (ex : for two accesses controller bus and network 2 simultaneous accesses can be sustained).

Si InterConnect = BUSSi InterConnect = RéseauGestion

des conflits

Gestion des

conflits

Gestion des

conflits

15 /15 /



DETERMINISM IN EMBEDDED AIRCRAFT SYSTEMS Abstract notion partially described in DO-297 Definition based on

Execution Integrity “Demonstrate that the Embedded Aircraft System mode during non-faulty

software execution remains nominal or degraded into an acceptable state”

Accumulate sufficient knowledge on the processor’s internal mechanisms.

WCET analysis

Platform Usage Domain More or Less difficult to analysis regarding the Airborne Software knowledge.

Robust Partitioning (not only for IMA system) Ensure by HW mechanism

Ensure by Operating System

Ensure at Airborne Software

16 /16 /


Multicore COTS Processors Conflicts Management

Spatial Management: how to manage accesses to be sure that one core can’t access to a space reserved for another core.

Temporal Management: How to manage accesses done by one core to all shared resources

(Memories, I/O, etc.) to be sure that accesses can be limited in time whatever activities of other core are (normal or abnormal).

Upper bound will be used for WCET computationFor Memory Accesses

Spatial Management is done by MMU and IOMMU (when existing) Temporal Management is more complex linked to interconnect (transaction

management), Memory Controller and Memory (transaction realization).

Operating SystemArchitecture Choice regarding Industry needs

Computer Number Reduction with low impact on legacy application AMP

Application Performance Improvement SMP


17 /17 /


MULTI-COREProcessor Selection

18 /18 /


Processor Selection: Selection Criteria

Selection criteria regarding the manufacturer situation Manufacturer has experience in the avionic domain Manufacturer is involved in the certification process Manufacturer publishes specific communications Manufacturer has a sufficient life expectancy Manufacturer ensures a long term support

Selection criteria regarding the Manufacturer openness regarding design and tests information

Design information on a COTS processor is mandatory to certify an avionic platform

Strong impact on the performance of the chip.

Il some manufacturers may not agree to communicate specific design information required to ensure determinism it is relevant to favor manufacturers who agree.

Moreover, for an avionic component, it is necessary to perform specific robustness tests, such as a SEE (Single Event Effect) or SER

19 /19 /



Focus on Architecture: Virtual Memory Management Virtual memory service (Memory Management Unit).

Translating virtual addresses into physical addresses, Verifying that the requesting software has the sufficient access rights. Multicore platforms VMM can be located at core, at processor or at both levels.

MMU components: Addresses translator and access rights checker. Storage device, Translation Look aside Buffers (TLB) to save locally the address

translation rules. Virtual memory is defined with pages frames (size & offset).

Focus on Architecture: Private cache & Scratchpad Use of hierarchical memory (caches and scratchpads) improves the

performance of software. Scratchpad usually viewed as a cache with its management implemented by

software. In a general way, timing variability when accessing private caches and

scratchpads is considered to be bounded. Content prediction depends on the cache replacement policy.

20 /20 /



Focus on HW assists for Debug & Monitoring

COTS processors provide debug mechanisms that enable breakpoint insertion, single step execution

Usual way to debug bare metal software is to use the JTAG interface.

On top of an operating system, debuggers such as GDB can be used.

21 /21 /


MULTI-CORERegarding Certification

22 /22 /


Multi-Core Processor features: INTERCONNECT

INTERCONNECT Overview

Interconnect, the first shared resource between cores. Interleaves the concurrent transactions sent by the cores to the shared

resources like caches, memories and I/O mapped in the address space. Its architecture has a strong impact on determinism and ensuring partitioning

insurance, and on the complexity of worst case analyses. Interconnect usually implements the following services:

Arbitration of incoming requests. This stage depends on several parameters: Arbitration rules Arbiter internal logic Network topology

Allocation of the physical destination devices when they are duplicated. For example when there is more than one MEMORY controllers.

Allocation of a path to the destination. When several paths exist between the source and the destination (depends

on routing rules). Support for atomic operations, hardware locking mechanisms Snooping mechanisms for cache coherency Inter Processors Interruptions (IPI) for inter-core communications

23 /23 /


Multi-Core Processor features: SHARED CACHE

SHARED CACHE Use of a shared cache in Embedded Aircraft Systems requires a solution to the

following problems: Shared cache content prediction. WCET calculability and robust partitioning requirements. Cache content integrity. Take care of SEU/MBU. Concurrent accesses impact. Potential restrictions on concurrent accesses to shared cache

have to appear in the Interconnect Usage Domain in the same way as concurrent accesses to shared memory.

Cache organizations Fully associative: Each memory row may be stored anywhere in the cache. N-way set associative cache: Each memory row may be stored in any way of some specific

sets of cache lines. Direct mapped cache: Each memory row may be stored in a single cache line.

Classic replacement policies are: Least Recently Used Pseudo Least Recently Used: Most Recently Used First In First Out Random

24 /24 /


Multi-Core Processor features: impact on Determinism

CACHE COHERENCY MECHANISM Required in architecture that integrates several storage devices

hosting one same data. Two families of coherency protocols:

Invalidate protocols: Accessed cache line is marked as invalidated in all locations.

Further accesses will miss and require a load to the main memory.

Class of protocols easier to implement and offers better performances.

Update protocols: Accessed cache line is updated.

Update request is broadcasted to all nodes : the ones containing the cache line are automatically updated.

Benefit: cache access will always hit without requesting the interconnect, thus traffic on the interconnect may be easier to control.

25 /25 /


Multi-Core Processor features: SHARED SERVICES

SHARED SERVICES Airborne Embedded Equipment is in charge of providing shared

services among the cores. Shared services:

Interrupts generation and routing to cores Core and processor clock configurations Timer configurations Watchdog configurations Power supply and reset Support for atomic operations

26 /26 /


Multi-Core Processor features: CORES

CORES The cores support the execution of multiple software instances in

parallel. They interact within two mechanisms:

Inter-core interrupts Shared memory

In the Embedded Aircraft Systems context, the use of inter-core interrupts (point-to-point or broadcast) might be the same as any external interrupt. It is acceptable under some conditions including (but not restricted to):

As a protection mechanism (a core can interrupt another core if it detects a faulty execution inside it)

When the destination core is actively waiting for being interrupted.

Memory mapping defined in the Memory Management Unit. Multi-core platforms embed one MMU per core. T Memory mapping definition is distributed among the cores. This raises the feature of coherency maintenance between all MMU.

A non-coherent configuration may weaken Robust Partitioning.

27 /27 /


Multi-Core Processor features: PERIPHERALS

PERIPHERALS: MAIN MEMORY AND I/O’S Sharing the main memory means sharing the physical storage

resources and the memory controllers. Storage resource can be partitioned when necessary: (space partitioning).

Sharing accesses to the memory controllers may in some cases increase the timing variability of a transaction with a factor higher than the number of accessing masters.

Shared I/O features are similar to shared services configuration: Access simultaneously read and/or write buffers.

Classic rules of time and space partitioning can apply: when it is not possible ensure that concurrent accesses will occur in disjoint time windows.

Initiate specific protocols operations: uninterrupted access is required during the protocol execution to be able to fulfill correctly the concerned protocol.

Like shared services, concurrent accesses to shared I/O may occur simultaneously from different cores.

Some I/O are accessed according to a protocol, others are accessed from a read and/or write buffer Atomic access patterns have to be ensured.

28 /28 /


MULTI-CORESoftware Aspects

29 /29 /


Multitasks scheduling features

Classic approach for a multitasked system is the hierarchical model based on processes (or partition) and threads

In ARINC 653, equivalent components are partitions and processes). Processes (or partitions) are executed from isolated memory areas. Inside a process, one or more threads are executed in the same address space.

Parallel programming models include two kinds of tasks: periodic and sporadic.

Processes and threads activation depends on a scheduling algorithm.

For an Embedded Aircraft Systems system, a scheduling algorithm shall verify the following properties:

Feasibility: Predictability:

Critical property ensuring that a set of tasks will meet its deadline.

Pre-emptive and priority based scheduling algorithms are preferred for single-core processors

30 /30 /


Airborne Software migration from single-core to multi-core

Porting multitasked Airborne Software from a single-core to a multi-core platform, required:

Airborne Software execution will still be correct Worst Case Execution Time will be calculated for each task or process.

Multitasked airborne software may not be efficiently executed on a multi-core platform if its tasks have dependencies requiring a specific execution order.

Care has to be taken if the Airborne Software is implemented within a cooperative tasks model.

Such an implementation usually removes protections in critical sections accesses.

In multi-core execution, critical section might be executed in parallel by different tasks, resulting in an erroneous execution critical section requires semaphore protection

31 /31 /


Partitioned system features

The most “flexible” component is the integration software layer. Possible designs:

A single OS instance shared among all the cores

A private OS instance per core

A virtualization layer hosting several operating systems in dedicated virtual machines.

Components evolution to take benefit of multi-core platforms

Partition Deployment One partition is activated on all cores and has an exclusive access to platform

resources

Symmetrical Multi-processing (SMP).

Each partition are activated on one core with true parallelism between partitions

Asymmetrical Multi-processing (AMP).

32 /32 /


Operating System global view

From Single Core to Multi-Core in AMP (Asymmetric multi-processing)

CORE

BRIDGE

Memory Controller

I/OController

BUS / NetworkInterface

Space & Time Partitionning

Operating System

CORE

INTERCONNECT

Memory Controller

I/OController


Operating System

CORE

Operating System

Space & Time Partitionning Space & Time Partitionning

APP1

T1

T2

T3

T4

APP2

T1

T2

T3

APP3

T1

T2

T3

T4

T5

Memory Controller

SolveConflict

This is a first step represented by this two cores processor example, for more than two cores (or less than two Memory Controller) we have to manage the conflicts to the Memory Controller.

33 /33 /


Operating System global view

From Single Core to Multi-Core in SMP (Symmetric multi-processing)

CORE

BRIDGE

Memory Controller

I/OController



Operating System

CORE

INTERCONNECT

Memory Controller

I/OController


Operating System

CORE


APP2

T1

T2

T3

APP3

T1

T2

T3

T4

T5

Memory Controller

SolveConflict

APP1

T1

T2

T3

T4

APP1

T1

T2

T3

T4

This is a first step represented by this two cores processor example, for more than two cores (or less than two Memory Controller) we have to manage the conflicts to the Memory Controller.

34 /34 /


Current mono-core concept

timePartition 1 Partition 2 Partition 3 Partition 4

Cor

e

OS

T1

T2

T4

T1

T3

T1

T3

T2T1

T2

T4

T1

T3

T1

T2

T4

T3

T5Appli. 1

Appli. 2

Appli. 3

idle

T

T

T

Thread / Process

CORE

BRIDGE

Memory Controller

I/OController



Operating System

APP1

T1

T2

T3

T4

APP2

T1

T2

T3

APP3

T1

T2

T3

T4

T5

T1

35 /35 /


AMP

timePartition 1.1 Partition 1.2 Partition 1.3 Partition 1.4

Cor

e 2

Cor

e 1

OS

1

T1

T2

T3

T1

T3

T1

T2T3

T1

T2

T3

T1

T3

T1

T2

T4

T1

T3

OS

2

T1

T2

T4

T1

T3

T1

T2T3

T1

T2 T2

T1

T3

T1

T2

T4T5

T3

Appli 5

Appli 6

Appli 7

idle

T

T

T

Appli.2

Appli 3

Appli 4

T

T

T

Appli. 1 T

Thread / Process

Partition 1.1 Partition 2.2 Partition 2.3 Partition 2.4

CORE

INTERCONNECT

Memory Controller

I/OController


Operating System

CORE

Operating System

Space & Time Partitionning Space & Time Partitionning

Memory Controller

APP1T1

T2T3

T4

APP2T1

T2

T3

APP3T1

T2T3

T4

T5

APP4

T1

T2

T3

APP5

T1

T2

T3

APP5T1

T2

T3

T4

When AMP mode is selected, the Use of Hypervisor is recommended to master the behavior of the Interconnect Usage Domain

36 /36 /


SMP

Appli. 1

Appli. 2

Appli. 3

idle

T

T

T

T1T1

T3

T1T3 T1

T4

T1

T3

timePartition 1 Partition 2 Partition 3 Partition 4

Cor

e 1

Cor

e 2

T2

T4

T2 T2

T1T1

T3

T2

T4

OS

Thread / Process

CORE

INTERCONNECT

Memory Controller

I/OController


Operating System

CORE


Memory Controller

APP1

T1

T2

T3

T4

APP2

T1

T2

T3

APP3

T1T2

T3

T4

T5

T5

When SMP mode is selected, Processes, Threads or Tasks should be allocated to cores statically to achieve determinism

37 /37 /


MULTI-COREFailure Mitigation Means & COTS Relative Features

38 /38 /


Multi-Core: Failure Mitigation

FMEA and/or FFPA for a single or a multi-core processor is not achievable at processor level

Mitigation has to be provided, by the equipment provider, at board level where this processor is used

Software Error Rate SEE (Single Event Effect)

Measurements on SER are usually performed by the manufacturers on their own

Deep Sub Micronics

DSM has impact of long term reliability

39 /39 /


An example of Stress Tests

Stress test have been kept identical from generation to generation to be able to guarantee in the industrial grade a usable life compatible with Avionics Requirements

40 /40 /


CONCLUSION

41 /41 /


Complexity of Multi-Core Processors has increased over the past few years, while the level of demonstration for design assurance should remain at least the same as- or better than for COTS without such increment in complexity.

A COTS component remains a COTS component (features proprietary data from the COTS manufacturer).

Approaches: Access to additional data under agreements with the COTS manufacturer And/or mitigation of potential COTS faults or errors at board or equipment

level,

CONCLUSIONS

42 /42 /


CONCLUSIONS

In this report we put emphasis on specific Multi-Core features linked to Shared Resource Accesses like Memory, Bus, Network, Internal Registers, Clock Management, etc.

These features are the main differences between single-core and multi-core devices that have to be managed

At Airborne Software Level If Airborne Software behavior is well known and well managed, then by allocating

Airborne Software applications to cores, we can demonstrate the non-interaction between cores.

The interconnect behavior shall be well known and well managed

At Hypervisor level In this configuration, the Hypervisor is used to constraint the behavior of the

interconnect. These constraints reduce the global performance of the multi-core processor but offer determinism and so the global behavior can be demonstrated.

43 /43 /


COMPLEMENTARY INFORMATION

44 /44 /


Multi-core Processor Usage Domain

Definition, Validation and Verification of a Usage Domain (UD) for such highly complex COTS Multi-Core processors is required.

This approach is already known and offered by existing certification guidance for Complex and Highly Complex COTS.

One recommendation would be to distinguish between the UD rules related to segregation constraints (e.g. segregation between cores), from the UD rules related to local limitations (within a single core).


45 /45 /


Significant features Determining WCET, knowing the high variability of execution time,

the following step by step approach can be one of the solution to ensure the temporal deterministic behavior of processors; such an approach is also valid for multi-core processors:

Characterization of execution time jitters of the operating system services,

Determination of the Worst Case Execution Time (WCET) plus allowed margins,

Incorporated real-time monitoring of actual exec time versus allowed WCET,

Collect data for assessment of the processor + Airborne Software operating behavior,

Depending on the above assessment, establish additional rules or limitations,

Apply necessary modifications


46 /46 /


Robust partitioning Mitigation to cater for the inherent complexity of multi-core processors

via functional robustness at Airborne Software level is possible whenever the developer has allowed access to- and detailed knowledge of- the computing platform.

Defensive programming techniques can be used to compensate for potential misbehaviors. This possibility is not accessible for Software execution platforms where Airborne Software developers have only access to an allocated portion of the platform with strict rules and requirements to meet in order to allow adequate operation of the whole integrated system.

Multi-software architectures are now common, hence robust partitioning of Airborne Software must then be ensured. For example an essential feature is the execution time variations due to jittering on partition switching that should be minimized to allow time-deterministic behavior. Indeed, guidance is that temporal determinism shall be ensured knowing given criteria..


Documents

Legal Entity/Division - Date Multicore For Avionics Certification Issue 2013 – 03 – 22