[original file is 20070831CELF_Busybox.pdf]
[translated by ikoma]

CELF Technical Jamboree #16
Aug. 31, 2007

OProfile Porting to MIPS Architecture

8/31/2007

Takehiko Nagano
Embedded System Platform Research Laboratory
Hitachi, Ltd.

OProfile Porting to MIPS Architecture

* Contents

Overview of OProfile
Background of porting OProfile
OProfile porting policy
OProfile supporting status
What were ported?
- Set hardware timer interrupt
- Implemented interrupt handler
- Modified system calls for event trace
- Others
Summary

Overview of Oprofile

* Tool to Measure System Performance

Systemwide profile, including kernel
- By using timer interrupt to sample program counter value, measures execution frequency of a program running on the system
Statistical results can be presented per binary image, per function, per address, etc.
- Can analyze libraries which started and/or stopped while profiling
Fully utilize various performance monitoring hardwares
- Collecting data such as cache misses, TLB misses

Overview of OProfile

[See original file for figure]

OProfile output example

root@tx4937:~# opreport --symbols
CPU: MIPS TX4937, speed 0 MHz (estimated)
Counted H_W_TIMER_2 events
(H_W_TIMER2_INTERRUPTS) with a unit mask of 0x00 (No
unit mask) count 100000
samples % app name symbol name
14685 96.1627 vmlinux r4k_wait
92 0.6024 vmlinux r4k_blast_icache
27 0.1768 vmlinux r4k_copy_page_d32
22 0.1441 vmlinux
r4k_blast_dcache_page
21 0.1375 vmlinux r4k_blast_dcache
20 0.1310 vmlinux ne_block_input
15 0.0982 bash internal_malloc
12 0.0786 ld-2.3.2.so _dl_lookup_symbol
11 0.0720 ld-2.3.2.so
_dl_lookup_versioned_symbol
11 0.0720 vmlinux serial_console_write
10 0.0655 vmlinux ne_block_output

+ - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - +
|                               +==============+                              |
                                |OProfile data||-----------\
|  +------------+               +-------------+             \                 |
   |kernel image|                     ^                      V
|  +------------+                     |               +----------------+      |
   +------------+    func      +---------------+      |    OProfile    |   
|  | app image  |    addr      |OProfile daemon|<---- |control/analysis|      |
   +------------+              +---------------+      |    commands    |
|                                                     +----------------+      |
      ==================
+ - // /proc/pid/maps  /- - - - - - - - - - - - - - - - - - - - - - - - - - - +
+--//                 /-------------------------------------------------------+
|   ------------------                                          Linux Kernel  |
|  +---+                     +----------------+       +-----+                 |
|  |pid|======sampling======>|OProfile modules|<=====>|Buffer|                |
|  +---+                     +----------------+       +-----+                 |
|                        ^        ^                                           |
+----------------------     sampling   ---------------------------------------+
+-----------------------||--------|-------------------------------------------+
|                       ||   +----|--------+                         CPU      |
|                            |    timer    |                                  |
|                +--------+  | +---------+ |                                  |
|                |PC value|  | |interrupt| |                                  |
|                +--------+  | +---------+ |                                  |
|                            +-------------+                                  |
|                                                                             |
+-----------------------------------------------------------------------------+

              Figure 1. Configuration and output example of OProfile

Background of Porting OProfile

* No Profiler Available which Satisfies Embedded System Developers' Needs

Systemwide profile is not available
- Want to analyze performance of multiple collaborating processes/threads
System perfonamance analysis is not available when processes switch in short time
- Sampling rate of gprof is 100Hz (in case of MVL3.1)
Performance analysis per thread is not available
==> OProfile seems able to solve these issues

Porting Policy of OProfile

* Port only basic functions of OProfile

It's OK if systemwide profiling becomes possible
- Thread level analysis already supported

* Higher sampling rate

Sampling by tick is inadequate for performance analysis of embedded systems
- Context switch occurs in tens of microsecond

OProflie Supporting Status

* OProfile Availability

Our target is TX4937 (with MVL3.1 kernel) -- not supported

+----------+----------+----------+----------+----------+----------+
|          |   i386   |  x86-64  |   arm    |    sh    |   mips   |
+----------+----------+----------+----------+----------+----------+
| Kernel   | athlon   | YES      | xscale   | SH7750S  | mips32   |
| 2.6.20   | p4       |          |          | SH7750   | mips64   |
|          | ppro     |          |          | SH7091   | R10000   |
|          | x86      |          |          |          | SB1      |
|          |          |          |          |          | RM9000   |
+----------+----------+----------+----------+----------+----------+
| Kernel   | YES      | YES      | NO       | NO       | mips64   |
| 2.4.x    |          |          |          |          | r5k      |
+----------+----------+----------+----------+----------+----------+
| Kernel   | YES      | YES      | NO       | NO       | NO       |
| 2.2.x    |          |          |          |          |          |
+----------+----------+----------+----------+----------+----------+
                               Embedded platforms have not been well supported
     Table 1. OProfile support status

What were Ported?

* Where work is required to port:

Interrupt handler
- Use hardware timer interrupt and get sampling rate higher
System calls (some of)
- Add tracing of symbol resolution information to system calls which control processes and libraries while profiling (execv etc.)
Others
- ==> Makefile, definition of events to capture

What were Ported?

* Source Code Required to Add or Modify

Kernel modules
- ==> Architecture dependent parts should be fixed, such as implementation of interrupt handlers and data capture at system call
  - /module/OProfile.c,OProfile.h,OProfile.c (timer processing etc. used in other architectures)
  - /module/mips/cpu_type.c,op_arch.h,op_mips_model.h,op_syscalls.c,op_timer.c (most important part in this fixing)
User land
- ==> CPU type, events to capture, etc.
  - /libop/op_cpu_type.c,cpu_type.h,op_events.h (specify CPU type etc.)
  - /events/events,unit_masks (fix for each event type)
  - /utils/opcontrol (script should be modified to support bash etc.)
  - And Makefile, configure script etc.

== What were Ported? Implementing Interrupt Handler==

* Interrupt Handler etc.

Timer initialization
Starting timer
Obtaining program counter
Halting timer

== What were Ported? Implementing Interrupt Handler==

* Timer Initialization and Start

Registers below should be set to initialize/start interrupt handler

+---+---------+--------+---------------------------------+-----+--------------------------+
| # |register |   item |       name                      |value| content                  |
+---+---------+--------+---------------------------------+-----+--------------------------+
| 1 | TMTCR2  | TCE    | Timer Counter Enable            |  1  | ON                       |
| 2 |         | CCDE   | Counter Clock Duration Enable   |  1  | ON                       |
| 3 |         | CRE    | Counter Reset Enable            |  1  | Reset                    |
| 4 |         | CCS    | Counter Clock Selection         |  1  | Use                      |
| 5 |         | TMODE  | Timer Mode                      |  0  | Use interval timer mode  |
| 6 | TMTISR2 |        |                                 |     |                          |
| 7 | TMCPRA2 |TMCCPRA2| Timer 2 Compare Register        |33000| Interrupt every 1msec    |
| 8 | TMITMR2 | TTIE   | Interval Timer IMCPRA interrupt |  1  | ON                       |
| 9 |         | TZCE   | Interval Timer Clear Permission |  1  | Allow                    |
+---+---------+--------+---------------------------------+-----+--------------------------+
   Table 2. Registers necessary for tiemer invocation (in case of 1 msec interrupt period)

== What were Ported? Implementing Interrupt Handler==

#define TX4937_TMR_SET(ofs,val) ( TX4927_WR32(TX4927_BASE + ofs , val) )
#define TX4937_TMR_RD(ofs) (TX4927_RD32(TX4927_BASE + ofs ))
static int timer_setup(void)
{
    TX4937_TMR_SET(TX4927_TMR2_TMTCR2, (0x1 << 7 | 0x1 << 6 | 0x1 << 5 | 0x0 << 2 |
 0x0 << 1 | 0x0 ));
    //TWIS,TPIBS,TPIAS and TIIS default
    TX4937_TMR_SET( TX4927_TMR2_TMTISR2, 0);
    TX4937_TMR_SET( TX4927_TMR2_TMCCDR2,0);
  //interrupt timing counter value ....
    TX4937_TMR_SET( TX4927_TMR2_TMCPRA2, 33000);
    return 0;
}
static void timer_start(void)
{
    u32 temp;

    //TTIE and TZCE are on
    TX4937_TMR_SET( TX4927_TMR2_TMITMR2 , 0x1 << 15 | 0x1 );
    temp = (TX4937_TMR_RD( TX4927_TMR2_TMTCR2 ) & ~(0x1 << 5));
    TX4937_TMR_SET( TX4927_TMR2_TMTCR2,temp );
    TX4937_TMR_SET( TX4927_TMR2_TMTCR2, temp | (0x1 << 7 ) );
        //counte clock divide
}

== What were Ported? Implementing Interrupt Handler==

* Obtaining Program Counter Value

Get symbols and relating information for running process
- Get cp0_epc value from pt_regs struct
- Get pid and tgid with current macro
- Reset timer counter of hardware timer (clear timer interrupt status register)

static void do_timer_interrupt(int irq, void * dev_id, struct pt_regs * regs)
{
    uint cpu = op_cpu_id();
    int usermode = user_mode(regs);
    /* Reset the interrupt */
    TX4937_TMR_SET( TX4927_TMR2_TMTISR2,0 );   // <===
    if ((sysctl.ctr[0].kernel && usermode)
        || (sysctl.ctr[0].user && !usermode))
        return;
    read_lock( &current->files->file_lock );
    op_do_profile(cpu, instruction_pointer(regs), IRQ_ENABLED(regs), 0);  // <===
    read_unlock( &current->files->file_lock );
}

What were Ported? Modifying System Calls

* Build /module/mips/op_syscall.c

Requried to collect information on processes and libraries which are newly executed while profiling

+---+---------------+----------------------------+------------------------------------------+
| # |name of        | function to get info       | note                                     |
|   | system call   |                            |                                          |
+---+---------------+----------------------------+------------------------------------------+
| 1 | clone         | oprof_report_fork          |                                          |
| 2 | execve        | oprof_output_map           | Register refferred is different          |
| 3 | exit          | oprof_put_note             |                                          |
| 4 | fork          | oprof_report_fork          |                                          |
| 5 | init_module   | oprof_put_note             |                                          |
| 6 | mmap          | oprof_output_map           | there is an interface as sys_old_mmap,   |
|   |               |                            | but it is internally do_mmap2;           |
|   |               |                            | interfaces for mips and x86 are not same |
| 7 | mmap2         | oprof_output_map           |                                          |
| 8 | vfork         | oprof_report_fork          | not used in mips kernel                  |
+---+---------------+----------------------------+------------------------------------------+
                     Table 3. System calls which requres to be fixed

What were Ported? Modifying System Calls

* How to Replace System Calls

Address of function implementing system call is registered, per system call number, in system_call_table (array) in call_table_t struct
==> When profiling, modify registered addresses for system calls to be fixed;
- Restore the addresses after profiling completed

What were Ported? Modifying System Calls

* e.g. Modificaiton of sys_execve

asmlinkage static int my_sys_execve(struct pt_regs regs)
{
    int error;
    char * filename;

    MOD_INC_USE_COUNT;

    filename = getname((char *) (long)regs.regs[4]);            // For modification per each architecture,
    error = PTR_ERR(filename);                                  // see /arch/*/kernel/syscall.c
    if (IS_ERR(filename))                                       //
        goto out;                                               //
    error = do_execve(filename, (char **) (long)regs.regs[5],   //
        (char **) (long)regs.regs[6], &regs);                   //

    //output info                        // For timing and contents of data aquisition,
    if (!error) {                        // borrow from the source code for other architecture
        PTRACE_OFF(current);             //
        oprof_output_maps(current);      // <-- after oprof_output_maps, information for symbol
    }                                    //      resolution need not be modified

    putname(filename);

out:
    MOD_DEC_USE_COUNT;
    return error;
}

What were Ported? Others

* Specify CPU type

Specify CPU name, folder of event file, etc. in /op_cpu_type.c

    { "NEC VR5432", "mips/vr5432", CPU_MIPS_VR5432, 2 },
    { "NEC VR5500", "mips/vr5500", CPU_MIPS_VR5500, 2 },
    { "MIPS TX4937", "mips/TX4937", CPU_MIPS_TX4937, 1},
    { "e500", "ppc/e500", CPU_PPC_E500, 4 },

* Build events file

Specify list of events in the folder specified as the second argument of /op_cpu_type.c
- e.g. name:H_W_TIMER_2 event:0xff counters:0 um:zero minimum:2 : H_W_TIMER2_INTERRUPTS

What were Ported? Others

問題発生 * Trouble breaks out!

In /kernel/ksyms.c, symbol of sys_call_table is not exported only for mips kernel
==> But don't want to touch kernel source code (or, correctly, not allowed to touch kernel)
==> Solved by extracting sys_call_table symbol from the built vmlinux, and giving the symbol information to the kenrel module

CFLAGS_op_syscalls.o += -DSYS_CALL_TABLE_ADDR=0x`$(CROSS_COMPILE)nm $(KSRC)/vmlinux
| grep sys_call_table | cut -f 1 -d ' '`

Summary

* Results

Ported OProfile onto TX4937
- Now systemwide profile is available
- Improved sampling rate using hardware timer interrupt
Remaining issues
- To fix configure script
- More events to support
- Feedback to the community (publishing source code)

END

OProfile Porting to MIPS Architecture

8/31/2007 Takehiko Nagano Embedded System Platform Research Laboratory Hitachi, Ltd.

HITACHI Inspire the Next