Hi Sandeep, Raghu It is good that the ARM support has confirmed the same. From my reading of the GICv3 specification, it says the read of ICC_IAR_ELx is self-synchronizing with respect to the interrupt it is activating. It does not mention that writes are self-synchronizing or whether it would be synchronized wrt to a Device memory access. So a dsb() is needed prior to EOI.
For GICv2, Device memory attribute only guarantees " the order of memory accesses arriving at a single peripheral of IMPLEMENTATION DEFINED size, as defined by the peripheral, must be the same order that occurs in a simple sequential execution of the program" (section B2.7.2 - Device memory - reordering, version E.a of ARM ARM). This means that a dmb() is needed if there has to be ordering enforced between peripherals.
If we take that GICv2 and the interrupting device are 2 peripherals, then there has to be a dmb() between them to ensure ordering between peripheral write and GIC_IAR write for EOI.
Best Regards Soby Mathew
-----Original Message----- From: Raghu Krishnamurthy raghu.ncstate@icloud.com Sent: 08 June 2020 16:16 To: Sandeep Tripathy sandeep.tripathy@broadcom.com; Soby Mathew Soby.Mathew@arm.com; Olivier Deprez Olivier.Deprez@arm.com; tf- a@lists.trustedfirmware.org Subject: Re: [TF-A] GICV3: system interface EOI ordering RFC
Thanks Sandeep. Good enough for me! Very interesting that it is seemingly not observed more often on real hardware.
Thanks -Raghu
On 6/8/20 8:11 AM, Sandeep Tripathy wrote:
Hi Raghu,
-----Original Message----- From: TF-A [mailto:tf-a-bounces@lists.trustedfirmware.org] On Behalf Of Raghu Krishnamurthy via TF-A Sent: Monday, June 08, 2020 7:09 PM To: tf-a@lists.trustedfirmware.org Subject: Re: [TF-A] GICV3: system interface EOI ordering RFC
Hello,
Sorry for chiming in late. Are we sure that the msr/mrs to the GICv3 EOI register is not a context synchronizing event or ensures ordering itself ? Also did the addition of DSB() fix the issue ? If yes, can you try adding a delay or a few NOP's/dummy instructions or an ISB to see if the issue goes away?
Yes these experiments were done to observe the behavior which led to this change. A definitive confirmation of how core treats these accesses can be only be from arm core team or by actually probing the transfers. I had a response from arm support on the very same question which reads as follows: "In theory, the memory barrier should be placed between the clearing of the peripheral and the access to the EOI register". In a slightly unrelated case I had observed this with NVIC on axi probe and there is no feasibility to repeat this with GICv3 at the moment.
We need to know the right thing to do architecturally and confirm that the issue is really due to reordering at the core before adding the DSB. Adding a DSB might be masking a timing issue.
Is the above response and what Soby had confirmed sufficient ? or could someone from arm get this reconfirmed from core design unless it is obvious to them ? And this is equally applicable to linux as well.
-Raghu
Thanks Sandeep
On 6/8/20 4:47 AM, Sandeep Tripathy via TF-A wrote:
-----Original Message----- From: Soby Mathew [mailto:Soby.Mathew@arm.com] Sent: Monday, June 08, 2020 3:36 PM To: Sandeep Tripathy; Olivier Deprez; tf-a@lists.trustedfirmware.org Cc: nd Subject: RE: [TF-A] GICV3: system interface EOI ordering RFC
Just some inputs from my side: I agree we need at least a dsb() after the write to MMIO region to silence the peripheral and before the EOI at GIC sysreg interface. Adding it to the GIC EOI
Thanks for the confirmation. Now it’s about choosing the right place.
API seems the logical thing to do but as Olivier mentions, there are interrupt handler which do not deal with MMIO (eg: the Systimer interrupt) so adding it to GICv3 API might be arduous for such handlers.
So there is a choice here to let the interrupt handlers to deal with ensuring completeness before EOI at sysreg interface or adding it to GICv3 EOI API and take the overhead for interrupt handlers which do not have to deal with MMIO.
Yes I feel either of these is a must to guarantee functionality architecturally though both approach end up with some unnecessary overhead.
If GICv3 api takes care then it is an overhead for some ISRs dealing with non-MMIO. At present I do not see an active use case in reference implementation where sys timer ISR is in a performance intensive path where one additional DSB will be perceivable. But there may be some I could be totally wrong in this assumption
(pmu/debug
or.. not sure). Whereas I can certainly imagine some MMIO ISRs in performance critical path where unnecessary DSB is not acceptable at all.
If the interrupt handler needs to ensure then it will generically add 'DSB' as I think the driver cannot and should not make assumptions about how EOI is done afterwards. That will be overhead for the ISRs dealing with MMIO peripherals and non GIC-v3. If we consider only GICv3+ then good. Otherwise I would prefer the 'plat_ic_end_of_interrupt' like Olivier mentioned with a #if GICv3 instead of each ISRs dealing with it.
The GICv3 legacy MMIO CPU interface is deprecated for TF-A and the sys interface is the only one GICv3 driver in TF-A supports.
Right we can ignore the GICv3 legacy mode but GICv2 will still remain ?
Best Regards Soby Mathew
Thanks Sandeep
-----Original Message----- From: TF-A tf-a-bounces@lists.trustedfirmware.org On Behalf Of
Sandeep
Tripathy via TF-A Sent: 08 June 2020 10:34 To: Olivier Deprez Olivier.Deprez@arm.com; tf-
a@lists.trustedfirmware.org
Subject: Re: [TF-A] GICV3: system interface EOI ordering RFC
Hi Olivier, > -----Original Message----- > From: Olivier Deprez [mailto:Olivier.Deprez@arm.com] > Sent: Monday, June 08, 2020 1:14 PM > To: tf-a@lists.trustedfirmware.org; Sandeep Tripathy > Subject: Re: [TF-A] GICV3: system interface EOI ordering RFC > > Hi Sandeep, > > gicv3_end_of_interrupt_sel1 is a static access helper macro. Its > naming precisely tells what it does at gicv3 module level. It is > called from the weak > plat_ic_end_of_interrupt function for BL32 image. > > I tend to think it is the driver responsibility to ensure the > module interrupt > acknowledge register write is reaching HW in order (or "be > visible to all > observers").
The driver should be agnostic of what interrupt controller is used and its behavior. And since 'all' writes were to mmio ranges mapped Device(nGnRE) /strongly-ordered(nGnRnE) there was no such need. This is a special case for GICv3 system interface only. Adding at driver level a DSB is unnecessary for other interrupt controllers.
> Also I suspect adding a dsb might not be required generically for > all kind of IP.
Here are you referring to the peripheral IP or interrupt controller IP ? The issue is reordering at arm core itself (STR to device address Vs msr(sysreg write)). So I think Its applicable for all IP.
> Adding a barrier in generic code might incur an unnecessary > bottleneck.
But if there is a need to *ensure* presence of at least one DSB between the write transfer from core to device clear and gicv3 eoi icc register write in a generic way then what other option we have. > > Thus wouldn't it be better to add the barrier to the overridden > platform function rather than in generic gicv3 code?
That can be done but I feel this is more to do with gicv3 system interface only. Inside plat_xxx one has to check #if GICV3 ...and system interface. > > I have a put a comment in the review, we can continue the > discussion there. > > Regards, > Olivier. > Thanks Sandeep > ________________________________________ > From: TF-A tf-a-bounces@lists.trustedfirmware.org on behalf of > Sandeep Tripathy via TF-A tf-a@lists.trustedfirmware.org > Sent: 05 June 2020 19:43 > To: tf-a@lists.trustedfirmware.org > Subject: [TF-A] GICV3: system interface EOI ordering RFC > > Hi, > In a typical interrupt handling sequence we do 1-Read IAR > 2-Do interrupt handling > 3-Clear the interrupt at source , so that the device > de-asserts IRQ request to > GIC > >> I am suggesting a need of DSB here in case of GICv3 +. > 4-Write to GIC cpu interface to do EOI. > > Till GICv2 and with GICv3 legacy interface ICC_EOI write is a > write over AMBA > interface. The > Addresses are mapped with (nR) attribute. Hence the write > transfers from the > core will be > generated at step 3 and 4 in order. Please ignore the > additional buffers/bridges > in path from > core till peripheral which has to be dealt separately as per SOC. > > Query: I understand GICv3 system interface accesses are not over > this protocol > and core will not > follow the ordering rule ? > > I have observed spurious interrupt issue/manifestation ( I don't > have the > transfers probed) in > RTOS environment where I have a primitive GICv3 driver but I > wonder why things does not fail in Linux or tf-a. If it is > working because from step(3) to step(4) we have > barriers by chace > due to other device register writes then I would suggest to have > one in the EOI > clearing API itself. > > RFC: https://review.trustedfirmware.org/c/TF-A/trusted-firmware-a/+/445 4 > > Thanks
> Sandeep
TF-A mailing list TF-A@lists.trustedfirmware.org https://lists.trustedfirmware.org/mailman/listinfo/tf-a
-- TF-A mailing list TF-A@lists.trustedfirmware.org https://lists.trustedfirmware.org/mailman/listinfo/tf-a