Introduction
Hook into ftrace TP mali/mali_job_slots_event to see which process has scheduled work on any given slot_id (e.g. Fragment) on the GPU. Basically a TOP for the ARM Mali GPU. There are Mali GPU counter that can tell you the overall utilization but this is not enough for observability.
There is a Mali trace point one can use
c
/*
* mali_job_slots_event - Reports change of job slot status.
* @gpu_id: Kbase device id
* @event_id: ORed together bitfields representing a type of event,
* made with the GATOR_MAKE_EVENT() macro.
*/
TRACE_EVENT(mali_job_slots_event, TP_PROTO(u32 gpu_id, u32 event_id, u32 tgid, u32 pid, u8 job_id),
TP_ARGS(gpu_id, event_id, tgid, pid, job_id),
TP_STRUCT__entry(__field(u32, gpu_id) __field(u32, event_id) __field(u32, tgid)
__field(u32, pid) __field(u8, job_id)),
TP_fast_assign(__entry->gpu_id = gpu_id; __entry->event_id = event_id;
__entry->tgid = tgid; __entry->pid = pid; __entry->job_id = job_id;),
TP_printk("gpu=%u event=%u tgid=%u pid=%u job_id=%u", __entry->gpu_id,
__entry->event_id, __entry->tgid, __entry->pid, __entry->job_id));
(https://android.googlesource.com/kernel/google-modules/gpu/+/refs/heads/android-gs-shusky-6.1-android16/borr_mali_kbase/mali_linux_trace.h#34)
There are start and end events (event_id) for the same TP. Start and end events can't be correlated by any field, because finish events are always {tgid: 0, pid: 0, job_id: 0}. I am not quite sure whether this is intentional or not, especially that the job_id is 0 for finish events.
eBPF
SEC("tracepoint/mali/mali_job_slots_event")
int trace_mali_job_slots(struct mali_job_slots_event_ctx *ctx)
Issue
After some time my accounting somehow broke. It looked like some end events were not accounted for.
Therefore, it was decided to start the eBPF program with some instrumentation and also collect an FTRACE trace via the Linux FTRACE sub-system in order to check whether it is a driver issue.
As a matter of fact, the Linux FTRACE events were all accounted for. Meaning, the start and finish events were in balance (e.g. start: 1000, end: 1000). But, the eBPF TP hits were not in balance (e.g. start: 1000, end 997).
NOTE: start and end have to be in check, otherwise I would always see 100% utilisation
I gave 'the machine' the task to check the BPF Kernel sources what the issue could be. And it came back with an explanation: bpf_prog_active.
You've hit on the exact reason why bpf_prog_active exists and why it's per-CPU. The answer is interrupts, softirqs, and NMIs — all of which run on the same CPU and can preempt the currently executing BPF program.
So, it seems structured TP's have some check that re-entry is disallowed on the same CPU in order to protect against recursion.
It seems, if an eBPF program was running on an CPU and an interrupt (finish event) came in triggering another TP hit, thus running the eBPF again, it would detect that another eBPF program is already running and just decided to not execute it SILENTLY.
The solution was to switch to RAW_TRACEPOINT, which work because it does not have this bpf_prog_active safety rail-guard but on the other hand, the Anrdoid BpfLoader disallows RAW_TRACEPOINTs, that is quite a shame :(
Question
Did anyone experience such an issue? Let me tell you, that finding this issue without KI would have been impossible.
Addendun
Android 14, arm64, Kernel 5.15
