Hi, Jens,
>The first is the descriptor passed with FFA_MEM_RETRIEVE_REQ_32,
>this one is not likely to need fragmentation support soon.
Could you tell me why? I'm a little unclear.
Could you describe the difference between the retrieve process you mean and FFA-spec figure 18.4 in detail?
I need to make sure that I really understand what you mean,
so that I can implement the missing support for fragmented mem retrieve response as you said.
Or do you mean that we just need to implement the action done by optee after it receives FFA_MEM_RETRIEVE_RESP_32?
Regards,
Yuye.
------------------------------------------------------------------
发件人：Jens Wiklander <jens.wiklander(a)linaro.org>
发送时间：2023年4月14日(星期五) 14:23
收件人：梅建强(禹夜) <meijianqiang.mjq(a)alibaba-inc.com>
抄 送：Olivier Deprez <Olivier.Deprez(a)arm.com>; hafnium <hafnium(a)lists.trustedfirmware.org>; op-tee <op-tee(a)lists.trustedfirmware.org>
主 题：Re: fragment transmission while retrieving memory
Hi Yuye,
On Fri, Apr 14, 2023 at 4:29 AM 梅建强(禹夜)
<meijianqiang.mjq(a)alibaba-inc.com> wrote:
>
>
> Hi, Jens, Olivier,
>
> If I understand properly, for this specific issue both optee and hafnium need to do further fix.
> The following discussion and questions are based on figure 18.4 in the FFA spec.
>
> As I saw in the optee code, the retrieve mechanism would start with this code:
> retrieve_desc = spmc_retrieve_req(cookie);
> And in the function spmc_retrieve_req,
> optee sets the total length to the same size as fragment0 length before sending FFA_MEM_FRAG_REQ to hafnium,
> which means hafnium will only retrieve the fragment0 memory region.
>
> In the example process,
> optee will determines the number of TX sized fragments of descriptor before sending FFA_MEM_FRAG_REQ to hafnium.
> I have a question here.
> How should optee calculate this quantity?
>
> Hafnium then receives the FFA_MEM_FRAG_REQ.
> As I saw in the hafnium code, hafnium blocks the case that total length>fragment length.
> if (fragment_length != length) {
> dlog_verbose("Fragmentation not yet supported.\n");
> return ffa_error(FFA_INVALID_PARAMETERS);
> }
> Obviously it is not what we expected.
> Then following the example process, hafnium should allocate handle and use it to associate fragments.
> I didn't find the corresponding implementation in hafnium code for this step.
> And I want to know how to implement the associate action here.
>
> After that, optee lacks the implementation of FFA_MEM_FRAG_RX and FFA_MEM_FRAG_TX ABI in CFG_CORE_SEL2_SPMC enable case.
> Is there any support plan for the implementation of fragmented memory retrieve in the optee community?
> Does the hafnium community have a plan to implement it cooperatively?
If I've understood this correctly we have two different cases of
eventual fragmentation of descriptors. The first is the descriptor
passed with FFA_MEM_RETRIEVE_REQ_32, this one is not likely to need
fragmentation support soon. The other is the descriptor received with
FFA_MEM_RETRIEVE_RESP, I believe this is where you've run into
trouble.
From OP-TEE point of view, it would make sense if you took lead on
implementing this. I can help with review etc.
Thanks,
Jens
>
> Thanks a lot for your support.
>
> Regards,
> Yuye.
>
>
>
>
>
> ------------------------------------------------------------------
> 发件人：Olivier Deprez <Olivier.Deprez(a)arm.com>
> 发送时间：2023年4月13日(星期四) 15:52
> 收件人：梅建强(禹夜) <meijianqiang.mjq(a)alibaba-inc.com>
> 抄 送：hafnium <hafnium(a)lists.trustedfirmware.org>; Jens Wiklander <jens.wiklander(a)linaro.org>
> 主 题：Re: fragment transmission while retrieving memory
>
> Hi Yuye, Jens,
>
> For the record, and If I understand properly the last comment in the github issue:
> OP-TEE is missing the implementation for receipt of a fragmented retrieve response
> There is no further fix to be done in Hafnium for this specific issue at this moment, please let me know otherwise.
>
> Thanks, Olivier.
>
>
> ________________________________
> From: 梅建强(禹夜) <meijianqiang.mjq(a)alibaba-inc.com>
> Sent: 12 April 2023 09:27
> To: Olivier Deprez <Olivier.Deprez(a)arm.com>
> Cc: hafnium <hafnium(a)lists.trustedfirmware.org>; Jens Wiklander <jens.wiklander(a)linaro.org>
> Subject: Re: fragment transmission while retrieving memory
>
>
>
>
> Hi, Olivier,
>
> In our setup,
> Hafnium commit: 997476a74571aec4f1a23590d45edf516f3934f4
> optee version: 3.20.0
> Thanks.
>
> Regards,
> Yuye.
>
> ------------------------------------------------------------------
> 发件人：Olivier Deprez <Olivier.Deprez(a)arm.com>
> 发送时间：2023年4月12日(星期三) 15:22
> 收件人：梅建强(禹夜) <meijianqiang.mjq(a)alibaba-inc.com>
> 抄 送：hafnium <hafnium(a)lists.trustedfirmware.org>; Jens Wiklander <jens.wiklander(a)linaro.org>
> 主 题：Re: fragment transmission while retrieving memory
>
> Hi Yuye,
>
> AFAIK concerning Hafnium, fragmented mem sharing is supported for FFA_MEM_SHARE/LEND/DONATE and FFA_MEM_RETRIEVE_RESP.
> (For the sake of clarity, this isn't supported for FFA_MEM_RETRIEVE_REQ but this shouldn't be a concern as this limitation only exists in the case of mem sharing to multiple borrowers. In your case of a single borrower the mem retrieve req. shouldn't have to be fragmented).
>
> Can you tell which hafnium commit hash is used in your setup?
>
> At the moment, I cannot tell if the issue described concern a miss in Hafnium or OP-TEE.
> I need to dig a bit further into both implementations and I'll let you know.
>
> Regards,
> Olivier.
>
>
> ________________________________
> From: 梅建强(禹夜) <meijianqiang.mjq(a)alibaba-inc.com>
> Sent: 12 April 2023 08:21
> To: Olivier Deprez <Olivier.Deprez(a)arm.com>
> Cc: hafnium <hafnium(a)lists.trustedfirmware.org>; Jens Wiklander <jens.wiklander(a)linaro.org>
> Subject: fragment transmission while retrieving memory
>
>
>
>
> Hi, Olivier,
>
> Recently, I've been working on this issue.
> https://github.com/OP-TEE/optee_os/issues/5943 <https://github.com/OP-TEE/optee_os/issues/5943 >
> Do you know any differences between hafnium's current implementation of fragment transmission while retrieving memory and the example process described in FF-A 1.1 Figure 18.4?
>
> Regards,
> Yuye.
>
>
Hi, Jens, Olivier,
update for optee:
https://github.com/OP-TEE/optee_os/pull/5966 <https://github.com/OP-TEE/optee_os/pull/5966 >
update for hafnium:
diff --git a/src/api.c b/src/api.c
index 5f699478..7e43e305 100644
--- a/src/api.c
+++ b/src/api.c
@@ -3191,7 +3191,8 @@ struct ffa_value api_ffa_mem_frag_rx(ffa_memory_handle_t handle,
return ffa_error(FFA_INVALID_PARAMETERS);
}
}
-
+ if (to->id==0x8001 && to->mailbox.state != MAILBOX_STATE_EMPTY)
+ to->mailbox.state = MAILBOX_STATE_EMPTY;
to_locked = vm_lock(to);
if (vm_is_mailbox_busy(to_locked)) {
diff --git a/src/ffa_memory.c b/src/ffa_memory.c
index 3e978d23..59e71ff3 100644
--- a/src/ffa_memory.c
+++ b/src/ffa_memory.c
@@ -2385,7 +2385,7 @@ struct ffa_value ffa_memory_retrieve_continue(struct vm_locked to_locked,
&fragment_length);
CHECK(remaining_constituent_count == 0);
to_locked.vm->mailbox.recv_size = fragment_length;
- to_locked.vm->mailbox.recv_sender = HF_HYPERVISOR_VM_ID;
+ to_locked.vm->mailbox.recv_sender = to_locked.vm->id;
to_locked.vm->mailbox.recv_func = FFA_MEM_FRAG_TX_32;
to_locked.vm->mailbox.state = MAILBOX_STATE_FULL;
It seems that to->mailbox.state needs to be set to MAILBOX_STATE_EMPTY firstly.
I have not figured out why, but large TA could be loaded successfully with above fix now.
Regards,
Yuye.
------------------------------------------------------------------
发件人：梅建强(禹夜) <meijianqiang.mjq(a)alibaba-inc.com>
发送时间：2023年4月17日(星期一) 02:16
收件人：Jens Wiklander <jens.wiklander(a)linaro.org>
抄 送：Olivier Deprez <Olivier.Deprez(a)arm.com>; hafnium <hafnium(a)lists.trustedfirmware.org>; op-tee <op-tee(a)lists.trustedfirmware.org>
主 题：fragment transmission while retrieving memory
Hi, Jens,
I have implement the missing support for fragmented mem retrieve response as you suggested.
I will push fix into the community code soon and please help review it later.
Thanks.
Regards,
Yuye.
------------------------------------------------------------------
发件人：Jens Wiklander <jens.wiklander(a)linaro.org>
发送时间：2023年4月14日(星期五) 14:23
收件人：梅建强(禹夜) <meijianqiang.mjq(a)alibaba-inc.com>
抄 送：Olivier Deprez <Olivier.Deprez(a)arm.com>; hafnium <hafnium(a)lists.trustedfirmware.org>; op-tee <op-tee(a)lists.trustedfirmware.org>
主 题：Re: fragment transmission while retrieving memory
Hi Yuye,
On Fri, Apr 14, 2023 at 4:29 AM 梅建强(禹夜)
<meijianqiang.mjq(a)alibaba-inc.com> wrote:
>
>
> Hi, Jens, Olivier,
>
> If I understand properly, for this specific issue both optee and hafnium need to do further fix.
> The following discussion and questions are based on figure 18.4 in the FFA spec.
>
> As I saw in the optee code, the retrieve mechanism would start with this code:
> retrieve_desc = spmc_retrieve_req(cookie);
> And in the function spmc_retrieve_req,
> optee sets the total length to the same size as fragment0 length before sending FFA_MEM_FRAG_REQ to hafnium,
> which means hafnium will only retrieve the fragment0 memory region.
>
> In the example process,
> optee will determines the number of TX sized fragments of descriptor before sending FFA_MEM_FRAG_REQ to hafnium.
> I have a question here.
> How should optee calculate this quantity?
>
> Hafnium then receives the FFA_MEM_FRAG_REQ.
> As I saw in the hafnium code, hafnium blocks the case that total length>fragment length.
> if (fragment_length != length) {
> dlog_verbose("Fragmentation not yet supported.\n");
> return ffa_error(FFA_INVALID_PARAMETERS);
> }
> Obviously it is not what we expected.
> Then following the example process, hafnium should allocate handle and use it to associate fragments.
> I didn't find the corresponding implementation in hafnium code for this step.
> And I want to know how to implement the associate action here.
>
> After that, optee lacks the implementation of FFA_MEM_FRAG_RX and FFA_MEM_FRAG_TX ABI in CFG_CORE_SEL2_SPMC enable case.
> Is there any support plan for the implementation of fragmented memory retrieve in the optee community?
> Does the hafnium community have a plan to implement it cooperatively?
If I've understood this correctly we have two different cases of
eventual fragmentation of descriptors. The first is the descriptor
passed with FFA_MEM_RETRIEVE_REQ_32, this one is not likely to need
fragmentation support soon. The other is the descriptor received with
FFA_MEM_RETRIEVE_RESP, I believe this is where you've run into
trouble.
From OP-TEE point of view, it would make sense if you took lead on
implementing this. I can help with review etc.
Thanks,
Jens
>
> Thanks a lot for your support.
>
> Regards,
> Yuye.
>
>
>
>
>
> ------------------------------------------------------------------
> 发件人：Olivier Deprez <Olivier.Deprez(a)arm.com>
> 发送时间：2023年4月13日(星期四) 15:52
> 收件人：梅建强(禹夜) <meijianqiang.mjq(a)alibaba-inc.com>
> 抄 送：hafnium <hafnium(a)lists.trustedfirmware.org>; Jens Wiklander <jens.wiklander(a)linaro.org>
> 主 题：Re: fragment transmission while retrieving memory
>
> Hi Yuye, Jens,
>
> For the record, and If I understand properly the last comment in the github issue:
> OP-TEE is missing the implementation for receipt of a fragmented retrieve response
> There is no further fix to be done in Hafnium for this specific issue at this moment, please let me know otherwise.
>
> Thanks, Olivier.
>
>
> ________________________________
> From: 梅建强(禹夜) <meijianqiang.mjq(a)alibaba-inc.com>
> Sent: 12 April 2023 09:27
> To: Olivier Deprez <Olivier.Deprez(a)arm.com>
> Cc: hafnium <hafnium(a)lists.trustedfirmware.org>; Jens Wiklander <jens.wiklander(a)linaro.org>
> Subject: Re: fragment transmission while retrieving memory
>
>
>
>
> Hi, Olivier,
>
> In our setup,
> Hafnium commit: 997476a74571aec4f1a23590d45edf516f3934f4
> optee version: 3.20.0
> Thanks.
>
> Regards,
> Yuye.
>
> ------------------------------------------------------------------
> 发件人：Olivier Deprez <Olivier.Deprez(a)arm.com>
> 发送时间：2023年4月12日(星期三) 15:22
> 收件人：梅建强(禹夜) <meijianqiang.mjq(a)alibaba-inc.com>
> 抄 送：hafnium <hafnium(a)lists.trustedfirmware.org>; Jens Wiklander <jens.wiklander(a)linaro.org>
> 主 题：Re: fragment transmission while retrieving memory
>
> Hi Yuye,
>
> AFAIK concerning Hafnium, fragmented mem sharing is supported for FFA_MEM_SHARE/LEND/DONATE and FFA_MEM_RETRIEVE_RESP.
> (For the sake of clarity, this isn't supported for FFA_MEM_RETRIEVE_REQ but this shouldn't be a concern as this limitation only exists in the case of mem sharing to multiple borrowers. In your case of a single borrower the mem retrieve req. shouldn't have to be fragmented).
>
> Can you tell which hafnium commit hash is used in your setup?
>
> At the moment, I cannot tell if the issue described concern a miss in Hafnium or OP-TEE.
> I need to dig a bit further into both implementations and I'll let you know.
>
> Regards,
> Olivier.
>
>
> ________________________________
> From: 梅建强(禹夜) <meijianqiang.mjq(a)alibaba-inc.com>
> Sent: 12 April 2023 08:21
> To: Olivier Deprez <Olivier.Deprez(a)arm.com>
> Cc: hafnium <hafnium(a)lists.trustedfirmware.org>; Jens Wiklander <jens.wiklander(a)linaro.org>
> Subject: fragment transmission while retrieving memory
>
>
>
>
> Hi, Olivier,
>
> Recently, I've been working on this issue.
> https://github.com/OP-TEE/optee_os/issues/5943 <https://github.com/OP-TEE/optee_os/issues/5943 >
> Do you know any differences between hafnium's current implementation of fragment transmission while retrieving memory and the example process described in FF-A 1.1 Figure 18.4?
>
> Regards,
> Yuye.
>
>
Hi, Jens,
I have implement the missing support for fragmented mem retrieve response as you suggested.
I will push fix into the community code soon and please help review it later.
Thanks.
Regards,
Yuye.
------------------------------------------------------------------
发件人：Jens Wiklander <jens.wiklander(a)linaro.org>
发送时间：2023年4月14日(星期五) 14:23
收件人：梅建强(禹夜) <meijianqiang.mjq(a)alibaba-inc.com>
抄 送：Olivier Deprez <Olivier.Deprez(a)arm.com>; hafnium <hafnium(a)lists.trustedfirmware.org>; op-tee <op-tee(a)lists.trustedfirmware.org>
主 题：Re: fragment transmission while retrieving memory
Hi Yuye,
On Fri, Apr 14, 2023 at 4:29 AM 梅建强(禹夜)
<meijianqiang.mjq(a)alibaba-inc.com> wrote:
>
>
> Hi, Jens, Olivier,
>
> If I understand properly, for this specific issue both optee and hafnium need to do further fix.
> The following discussion and questions are based on figure 18.4 in the FFA spec.
>
> As I saw in the optee code, the retrieve mechanism would start with this code:
> retrieve_desc = spmc_retrieve_req(cookie);
> And in the function spmc_retrieve_req,
> optee sets the total length to the same size as fragment0 length before sending FFA_MEM_FRAG_REQ to hafnium,
> which means hafnium will only retrieve the fragment0 memory region.
>
> In the example process,
> optee will determines the number of TX sized fragments of descriptor before sending FFA_MEM_FRAG_REQ to hafnium.
> I have a question here.
> How should optee calculate this quantity?
>
> Hafnium then receives the FFA_MEM_FRAG_REQ.
> As I saw in the hafnium code, hafnium blocks the case that total length>fragment length.
> if (fragment_length != length) {
> dlog_verbose("Fragmentation not yet supported.\n");
> return ffa_error(FFA_INVALID_PARAMETERS);
> }
> Obviously it is not what we expected.
> Then following the example process, hafnium should allocate handle and use it to associate fragments.
> I didn't find the corresponding implementation in hafnium code for this step.
> And I want to know how to implement the associate action here.
>
> After that, optee lacks the implementation of FFA_MEM_FRAG_RX and FFA_MEM_FRAG_TX ABI in CFG_CORE_SEL2_SPMC enable case.
> Is there any support plan for the implementation of fragmented memory retrieve in the optee community?
> Does the hafnium community have a plan to implement it cooperatively?
If I've understood this correctly we have two different cases of
eventual fragmentation of descriptors. The first is the descriptor
passed with FFA_MEM_RETRIEVE_REQ_32, this one is not likely to need
fragmentation support soon. The other is the descriptor received with
FFA_MEM_RETRIEVE_RESP, I believe this is where you've run into
trouble.
From OP-TEE point of view, it would make sense if you took lead on
implementing this. I can help with review etc.
Thanks,
Jens
>
> Thanks a lot for your support.
>
> Regards,
> Yuye.
>
>
>
>
>
> ------------------------------------------------------------------
> 发件人：Olivier Deprez <Olivier.Deprez(a)arm.com>
> 发送时间：2023年4月13日(星期四) 15:52
> 收件人：梅建强(禹夜) <meijianqiang.mjq(a)alibaba-inc.com>
> 抄 送：hafnium <hafnium(a)lists.trustedfirmware.org>; Jens Wiklander <jens.wiklander(a)linaro.org>
> 主 题：Re: fragment transmission while retrieving memory
>
> Hi Yuye, Jens,
>
> For the record, and If I understand properly the last comment in the github issue:
> OP-TEE is missing the implementation for receipt of a fragmented retrieve response
> There is no further fix to be done in Hafnium for this specific issue at this moment, please let me know otherwise.
>
> Thanks, Olivier.
>
>
> ________________________________
> From: 梅建强(禹夜) <meijianqiang.mjq(a)alibaba-inc.com>
> Sent: 12 April 2023 09:27
> To: Olivier Deprez <Olivier.Deprez(a)arm.com>
> Cc: hafnium <hafnium(a)lists.trustedfirmware.org>; Jens Wiklander <jens.wiklander(a)linaro.org>
> Subject: Re: fragment transmission while retrieving memory
>
>
>
>
> Hi, Olivier,
>
> In our setup,
> Hafnium commit: 997476a74571aec4f1a23590d45edf516f3934f4
> optee version: 3.20.0
> Thanks.
>
> Regards,
> Yuye.
>
> ------------------------------------------------------------------
> 发件人：Olivier Deprez <Olivier.Deprez(a)arm.com>
> 发送时间：2023年4月12日(星期三) 15:22
> 收件人：梅建强(禹夜) <meijianqiang.mjq(a)alibaba-inc.com>
> 抄 送：hafnium <hafnium(a)lists.trustedfirmware.org>; Jens Wiklander <jens.wiklander(a)linaro.org>
> 主 题：Re: fragment transmission while retrieving memory
>
> Hi Yuye,
>
> AFAIK concerning Hafnium, fragmented mem sharing is supported for FFA_MEM_SHARE/LEND/DONATE and FFA_MEM_RETRIEVE_RESP.
> (For the sake of clarity, this isn't supported for FFA_MEM_RETRIEVE_REQ but this shouldn't be a concern as this limitation only exists in the case of mem sharing to multiple borrowers. In your case of a single borrower the mem retrieve req. shouldn't have to be fragmented).
>
> Can you tell which hafnium commit hash is used in your setup?
>
> At the moment, I cannot tell if the issue described concern a miss in Hafnium or OP-TEE.
> I need to dig a bit further into both implementations and I'll let you know.
>
> Regards,
> Olivier.
>
>
> ________________________________
> From: 梅建强(禹夜) <meijianqiang.mjq(a)alibaba-inc.com>
> Sent: 12 April 2023 08:21
> To: Olivier Deprez <Olivier.Deprez(a)arm.com>
> Cc: hafnium <hafnium(a)lists.trustedfirmware.org>; Jens Wiklander <jens.wiklander(a)linaro.org>
> Subject: fragment transmission while retrieving memory
>
>
>
>
> Hi, Olivier,
>
> Recently, I've been working on this issue.
> https://github.com/OP-TEE/optee_os/issues/5943 <https://github.com/OP-TEE/optee_os/issues/5943 >
> Do you know any differences between hafnium's current implementation of fragment transmission while retrieving memory and the example process described in FF-A 1.1 Figure 18.4?
>
> Regards,
> Yuye.
>
>
Hi, Jens, Olivier,
Is it possible to solve this problem by increasing the buffer size such as 64k?
We plan to adopt this temporary scheme first and then implement the standard retrieve process later.
If this scheme is feasible, I have a few questions.
1. Should the RXTX buffer between tee driver and hafnium the same as the one between hafnium and optee?
2. Which FFA instance is currently allocating these buffers?
3. For Hafnium, I want to increase the HF_MAILBOX_SIZE in Hafnium, but there seem to be two restrictions:
static_assert(HF_MAILBOX_SIZE == PAGE_SIZE,
"Currently, a page is mapped for the send and receive buffers so "
"the maximum request is the size of a page.");
static_assert(MM_PPOOL_ENTRY_SIZE >= HF_MAILBOX_SIZE,
"The page pool entry size must be at least as big as the mailbox "
"size, so that memory region descriptors can be copied from the "
"mailbox for memory sharing.");
I tried to increase the MM_PPOOL_ENTRY_SIZE,
but optee doesn't seem to be booted properly.
Do you know how to avoid the impact of these restrictions?
Regards,
Yuye.
------------------------------------------------------------------
发件人：Jens Wiklander via Hafnium <hafnium(a)lists.trustedfirmware.org>
发送时间：2023年4月14日(星期五) 14:23
收件人：梅建强(禹夜) <meijianqiang.mjq(a)alibaba-inc.com>
抄 送：Olivier Deprez <Olivier.Deprez(a)arm.com>; hafnium <hafnium(a)lists.trustedfirmware.org>; op-tee <op-tee(a)lists.trustedfirmware.org>
主 题：[Hafnium] Re: fragment transmission while retrieving memory
Hi Yuye,
On Fri, Apr 14, 2023 at 4:29 AM 梅建强(禹夜)
<meijianqiang.mjq(a)alibaba-inc.com> wrote:
>
>
> Hi, Jens, Olivier,
>
> If I understand properly, for this specific issue both optee and hafnium need to do further fix.
> The following discussion and questions are based on figure 18.4 in the FFA spec.
>
> As I saw in the optee code, the retrieve mechanism would start with this code:
> retrieve_desc = spmc_retrieve_req(cookie);
> And in the function spmc_retrieve_req,
> optee sets the total length to the same size as fragment0 length before sending FFA_MEM_FRAG_REQ to hafnium,
> which means hafnium will only retrieve the fragment0 memory region.
>
> In the example process,
> optee will determines the number of TX sized fragments of descriptor before sending FFA_MEM_FRAG_REQ to hafnium.
> I have a question here.
> How should optee calculate this quantity?
>
> Hafnium then receives the FFA_MEM_FRAG_REQ.
> As I saw in the hafnium code, hafnium blocks the case that total length>fragment length.
> if (fragment_length != length) {
> dlog_verbose("Fragmentation not yet supported.\n");
> return ffa_error(FFA_INVALID_PARAMETERS);
> }
> Obviously it is not what we expected.
> Then following the example process, hafnium should allocate handle and use it to associate fragments.
> I didn't find the corresponding implementation in hafnium code for this step.
> And I want to know how to implement the associate action here.
>
> After that, optee lacks the implementation of FFA_MEM_FRAG_RX and FFA_MEM_FRAG_TX ABI in CFG_CORE_SEL2_SPMC enable case.
> Is there any support plan for the implementation of fragmented memory retrieve in the optee community?
> Does the hafnium community have a plan to implement it cooperatively?
If I've understood this correctly we have two different cases of
eventual fragmentation of descriptors. The first is the descriptor
passed with FFA_MEM_RETRIEVE_REQ_32, this one is not likely to need
fragmentation support soon. The other is the descriptor received with
FFA_MEM_RETRIEVE_RESP, I believe this is where you've run into
trouble.
From OP-TEE point of view, it would make sense if you took lead on
implementing this. I can help with review etc.
Thanks,
Jens
>
> Thanks a lot for your support.
>
> Regards,
> Yuye.
>
>
>
>
>
> ------------------------------------------------------------------
> 发件人：Olivier Deprez <Olivier.Deprez(a)arm.com>
> 发送时间：2023年4月13日(星期四) 15:52
> 收件人：梅建强(禹夜) <meijianqiang.mjq(a)alibaba-inc.com>
> 抄 送：hafnium <hafnium(a)lists.trustedfirmware.org>; Jens Wiklander <jens.wiklander(a)linaro.org>
> 主 题：Re: fragment transmission while retrieving memory
>
> Hi Yuye, Jens,
>
> For the record, and If I understand properly the last comment in the github issue:
> OP-TEE is missing the implementation for receipt of a fragmented retrieve response
> There is no further fix to be done in Hafnium for this specific issue at this moment, please let me know otherwise.
>
> Thanks, Olivier.
>
>
> ________________________________
> From: 梅建强(禹夜) <meijianqiang.mjq(a)alibaba-inc.com>
> Sent: 12 April 2023 09:27
> To: Olivier Deprez <Olivier.Deprez(a)arm.com>
> Cc: hafnium <hafnium(a)lists.trustedfirmware.org>; Jens Wiklander <jens.wiklander(a)linaro.org>
> Subject: Re: fragment transmission while retrieving memory
>
>
>
>
> Hi, Olivier,
>
> In our setup,
> Hafnium commit: 997476a74571aec4f1a23590d45edf516f3934f4
> optee version: 3.20.0
> Thanks.
>
> Regards,
> Yuye.
>
> ------------------------------------------------------------------
> 发件人：Olivier Deprez <Olivier.Deprez(a)arm.com>
> 发送时间：2023年4月12日(星期三) 15:22
> 收件人：梅建强(禹夜) <meijianqiang.mjq(a)alibaba-inc.com>
> 抄 送：hafnium <hafnium(a)lists.trustedfirmware.org>; Jens Wiklander <jens.wiklander(a)linaro.org>
> 主 题：Re: fragment transmission while retrieving memory
>
> Hi Yuye,
>
> AFAIK concerning Hafnium, fragmented mem sharing is supported for FFA_MEM_SHARE/LEND/DONATE and FFA_MEM_RETRIEVE_RESP.
> (For the sake of clarity, this isn't supported for FFA_MEM_RETRIEVE_REQ but this shouldn't be a concern as this limitation only exists in the case of mem sharing to multiple borrowers. In your case of a single borrower the mem retrieve req. shouldn't have to be fragmented).
>
> Can you tell which hafnium commit hash is used in your setup?
>
> At the moment, I cannot tell if the issue described concern a miss in Hafnium or OP-TEE.
> I need to dig a bit further into both implementations and I'll let you know.
>
> Regards,
> Olivier.
>
>
> ________________________________
> From: 梅建强(禹夜) <meijianqiang.mjq(a)alibaba-inc.com>
> Sent: 12 April 2023 08:21
> To: Olivier Deprez <Olivier.Deprez(a)arm.com>
> Cc: hafnium <hafnium(a)lists.trustedfirmware.org>; Jens Wiklander <jens.wiklander(a)linaro.org>
> Subject: fragment transmission while retrieving memory
>
>
>
>
> Hi, Olivier,
>
> Recently, I've been working on this issue.
> https://github.com/OP-TEE/optee_os/issues/5943 <https://github.com/OP-TEE/optee_os/issues/5943 >
> Do you know any differences between hafnium's current implementation of fragment transmission while retrieving memory and the example process described in FF-A 1.1 Figure 18.4?
>
> Regards,
> Yuye.
>
>
--
Hafnium mailing list -- hafnium(a)lists.trustedfirmware.org
To unsubscribe send an email to hafnium-leave(a)lists.trustedfirmware.org
Hi, Jens, Olivier,
If I understand properly, for this specific issue both optee and hafnium need to do further fix.
The following discussion and questions are based on figure 18.4 in the FFA spec.
As I saw in the optee code, the retrieve mechanism would start with this code:
retrieve_desc = spmc_retrieve_req(cookie);
And in the function spmc_retrieve_req,
optee sets the total length to the same size as fragment0 length before sending FFA_MEM_FRAG_REQ to hafnium,
which means hafnium will only retrieve the fragment0 memory region.
In the example process,
optee will determines the number of TX sized fragments of descriptor before sending FFA_MEM_FRAG_REQ to hafnium.
I have a question here.
How should optee calculate this quantity?
Hafnium then receives the FFA_MEM_FRAG_REQ.
As I saw in the hafnium code, hafnium blocks the case that total length>fragment length.
if (fragment_length != length) {
dlog_verbose("Fragmentation not yet supported.\n");
return ffa_error(FFA_INVALID_PARAMETERS);
}
Obviously it is not what we expected.
Then following the example process, hafnium should allocate handle and use it to associate fragments.
I didn't find the corresponding implementation in hafnium code for this step.
And I want to know how to implement the associate action here.
After that, optee lacks the implementation of FFA_MEM_FRAG_RX and FFA_MEM_FRAG_TX ABI in CFG_CORE_SEL2_SPMC enable case.
Is there any support plan for the implementation of fragmented memory retrieve in the optee community?
Does the hafnium community have a plan to implement it cooperatively?
Thanks a lot for your support.
Regards,
Yuye.
------------------------------------------------------------------
发件人：Olivier Deprez <Olivier.Deprez(a)arm.com>
发送时间：2023年4月13日(星期四) 15:52
收件人：梅建强(禹夜) <meijianqiang.mjq(a)alibaba-inc.com>
抄 送：hafnium <hafnium(a)lists.trustedfirmware.org>; Jens Wiklander <jens.wiklander(a)linaro.org>
主 题：Re: fragment transmission while retrieving memory
Hi Yuye, Jens,
For the record, and If I understand properly the last comment in the github issue:
OP-TEE is missing the implementation for receipt of a fragmented retrieve response
There is no further fix to be done in Hafnium for this specific issue at this moment, please let me know otherwise.
Thanks, Olivier.
From: 梅建强(禹夜) <meijianqiang.mjq(a)alibaba-inc.com>
Sent: 12 April 2023 09:27
To: Olivier Deprez <Olivier.Deprez(a)arm.com>
Cc: hafnium <hafnium(a)lists.trustedfirmware.org>; Jens Wiklander <jens.wiklander(a)linaro.org>
Subject: Re: fragment transmission while retrieving memory
Hi, Olivier,
In our setup,
Hafnium commit: 997476a74571aec4f1a23590d45edf516f3934f4
optee version: 3.20.0
Thanks.
Regards,
Yuye.
------------------------------------------------------------------
发件人：Olivier Deprez <Olivier.Deprez(a)arm.com>
发送时间：2023年4月12日(星期三) 15:22
收件人：梅建强(禹夜) <meijianqiang.mjq(a)alibaba-inc.com>
抄 送：hafnium <hafnium(a)lists.trustedfirmware.org>; Jens Wiklander <jens.wiklander(a)linaro.org>
主 题：Re: fragment transmission while retrieving memory
Hi Yuye,
AFAIK concerning Hafnium, fragmented mem sharing is supported for FFA_MEM_SHARE/LEND/DONATE and FFA_MEM_RETRIEVE_RESP.
(For the sake of clarity, this isn't supported for FFA_MEM_RETRIEVE_REQ but this shouldn't be a concern as this limitation only exists in the case of mem sharing to multiple borrowers. In your case of a single borrower the mem retrieve req. shouldn't have to be fragmented).
Can you tell which hafnium commit hash is used in your setup?
At the moment, I cannot tell if the issue described concern a miss in Hafnium or OP-TEE.
I need to dig a bit further into both implementations and I'll let you know.
Regards,
Olivier.
From: 梅建强(禹夜) <meijianqiang.mjq(a)alibaba-inc.com>
Sent: 12 April 2023 08:21
To: Olivier Deprez <Olivier.Deprez(a)arm.com>
Cc: hafnium <hafnium(a)lists.trustedfirmware.org>; Jens Wiklander <jens.wiklander(a)linaro.org>
Subject: fragment transmission while retrieving memory
Hi, Olivier,
Recently, I've been working on this issue.
https://github.com/OP-TEE/optee_os/issues/5943 <https://github.com/OP-TEE/optee_os/issues/5943 >
Do you know any differences between hafnium's current implementation of fragment transmission while retrieving memory and the example process described in FF-A 1.1 Figure 18.4?
Regards,
Yuye.
Hi, Olivier,
In our setup,
Hafnium commit: 997476a74571aec4f1a23590d45edf516f3934f4
optee version: 3.20.0
Thanks.
Regards,
Yuye.
------------------------------------------------------------------
发件人：Olivier Deprez <Olivier.Deprez(a)arm.com>
发送时间：2023年4月12日(星期三) 15:22
收件人：梅建强(禹夜) <meijianqiang.mjq(a)alibaba-inc.com>
抄 送：hafnium <hafnium(a)lists.trustedfirmware.org>; Jens Wiklander <jens.wiklander(a)linaro.org>
主 题：Re: fragment transmission while retrieving memory
Hi Yuye,
AFAIK concerning Hafnium, fragmented mem sharing is supported for FFA_MEM_SHARE/LEND/DONATE and FFA_MEM_RETRIEVE_RESP.
(For the sake of clarity, this isn't supported for FFA_MEM_RETRIEVE_REQ but this shouldn't be a concern as this limitation only exists in the case of mem sharing to multiple borrowers. In your case of a single borrower the mem retrieve req. shouldn't have to be fragmented).
Can you tell which hafnium commit hash is used in your setup?
At the moment, I cannot tell if the issue described concern a miss in Hafnium or OP-TEE.
I need to dig a bit further into both implementations and I'll let you know.
Regards,
Olivier.
From: 梅建强(禹夜) <meijianqiang.mjq(a)alibaba-inc.com>
Sent: 12 April 2023 08:21
To: Olivier Deprez <Olivier.Deprez(a)arm.com>
Cc: hafnium <hafnium(a)lists.trustedfirmware.org>; Jens Wiklander <jens.wiklander(a)linaro.org>
Subject: fragment transmission while retrieving memory
Hi, Olivier,
Recently, I've been working on this issue.
https://github.com/OP-TEE/optee_os/issues/5943 <https://github.com/OP-TEE/optee_os/issues/5943 >
Do you know any differences between hafnium's current implementation of fragment transmission while retrieving memory and the example process described in FF-A 1.1 Figure 18.4?
Regards,
Yuye.
Hi, Olivier,
Whether Boot Information Blob is a file similar to manifest.dtb or sp.pkg?
How do I generate this blob?
Does it need to be packaged into atf firmware at compile phase？
Thanks.
Regards,
Yuye.
------------------------------------------------------------------
发件人：Olivier Deprez <Olivier.Deprez(a)arm.com>
发送时间：2023年3月21日(星期二) 22:37
收件人：Jens Wiklander <jens.wiklander(a)linaro.org>
抄 送：hafnium(a)lists.trustedfirmware.org <hafnium(a)lists.trustedfirmware.org>; OP-TEE TrustedFirmware <op-tee(a)lists.trustedfirmware.org>
主 题：Re: [Hafnium] Boot arguments for S-EL1 with SPMC at S-EL2
Hi Jens,
See comments inline [OD].
Regards,
Olivier.
________________________________
From: Jens Wiklander via Hafnium <hafnium(a)lists.trustedfirmware.org>
Sent: 21 March 2023 09:30
To: Olivier Deprez <Olivier.Deprez(a)arm.com>
Cc: hafnium(a)lists.trustedfirmware.org <hafnium(a)lists.trustedfirmware.org>; OP-TEE TrustedFirmware <op-tee(a)lists.trustedfirmware.org>
Subject: [Hafnium] Boot arguments for S-EL1 with SPMC at S-EL2
Hi Olivier,
I'm trying to implement a relocatable OP-TEE binary so it can be
loaded at different physical addresses without the need to recompile
it. This means that in the case with Hafnium when changing
"load-address" or "entrypoint-offset" in the OP-TEE SP manifest
there's no need to recompile OP-TEE. For this to work OP-TEE must be
able to figure out which memory range it's supposed to reside in.
[OD] Fair enough, I believe that would conflate into some generic PIE support?
I can see how this helps with the multi OP-TEE instances use case, perhaps?
Currently, OP-TEE knows the entry point address from PC and "memory
size" from X0. However, the "memory size" is from the "load-address"
so "entrypoint-offset" must be subtracted from PC in order to know the
allocated memory range.
Do you have ideas on how OP-TEE at runtime can determine the allocated
memory range?
[OD] The standard way would be to use the FF-A boot protocol e.g. if you add this to OP-TEE's SP manifest:
+ /* Boot protocol */
+ gp-register-num = <0x0>;
+
+ /* Boot Info */
+ boot-info {
+ compatible = "arm,ffa-manifest-boot-info";
+ ffa_manifest;
+ };
See https://trustedfirmware-a.readthedocs.io/en/latest/components/secure-partit… <https://trustedfirmware-a.readthedocs.io/en/latest/components/secure-partit… >
At SP entry, x0 contains the address of a 'Boot Information Blob' (FF-A v1.1 REL0 section 5.4.2) starting with a Boot Information Header (Table 5.9).
The SP manifest DT address can be extracted from a Boot Information Descriptor (Table 5.8).
Eventually the SP is able to parse its own SP manifest, into which it finds FF-A standard fields (load-address...) in addition to any impdef field added.
Note the partition size is not standarded (it should probably be), so for starters you can add your own memsize field.
There is some sample usage in TF-a-tests:
https://git.trustedfirmware.org/TF-A/tf-a-tests.git/tree/spm/cactus/cactus_… <https://git.trustedfirmware.org/TF-A/tf-a-tests.git/tree/spm/cactus/cactus_… >
and also in context of EDKII/StMM:
https://github.com/odeprez/edk2/commit/7066c913cb227056d68ea99de5e92f2067fe… <https://github.com/odeprez/edk2/commit/7066c913cb227056d68ea99de5e92f2067fe… >
Thanks,
Jens
--
Hafnium mailing list -- hafnium(a)lists.trustedfirmware.org
To unsubscribe send an email to hafnium-leave(a)lists.trustedfirmware.org
Hi, experts,
We seem to have solved the problem. This problem is not caused by the load_address of optee, but by the security space configured in the ATF.
It seems that the security space we have configured is too large. And why will it causes the problem needs to be further identified.
Regards,
Yuye.
------------------------------------------------------------------
发件人：梅建强(禹夜) <meijianqiang.mjq(a)alibaba-inc.com>
发送时间：2023年3月22日(星期三) 11:50
收件人：Olivier Deprez <Olivier.Deprez(a)arm.com>; Jens Wiklander <jens.wiklander(a)linaro.org>
抄 送：hafnium <hafnium(a)lists.trustedfirmware.org>; op-tee <op-tee(a)lists.trustedfirmware.org>
主 题：load optee at 36bit address
Hi, experts,
We are currently moving optee to a 36bit address to boot with environment
that hafnium runs at sel2 as spmc and optee runs at sel1 as sp.
Now we have moved hafnium to 0x880000000 and run successfully.
Then we tried moving optee to a 36bit address (0x89000000) as well.
Although hafnium and optee were successfully initialized on the primary cpu,
psci_cpu_on does not seem to be entered into when the secondary cpu is started.
The error is as follows:
https://github.com/OP-TEE/optee_os/issues/5895 <https://github.com/OP-TEE/optee_os/issues/5895 >
Is there any difference between the two cases
where hafnium and optee initialize the secondary cpu with different load_address?
The log print shows that the secondary cpu has not entered hafnium.
Could Hafnium be affected by the 36bit address when dealing with psci related transactions?
Regards,
Yuye.
Hi, Olivier
I need to explain the query in more detail.
>For building/running hafnium , did you have to apply a change similar to:
>https://review.trustedfirmware.org/c/hafnium/hafnium/+/18510 <https://review.trustedfirmware.org/c/hafnium/hafnium/+/18510 >
Yes.
Some time ago, we used hafnium codebase based on the commit ca03054ba6a351534b93e6d64c12e671578eb340.
And I verified the fix. The compilation passed, but the system cannot boot successfully.
Later, we updated the hafnium codebase based on the commit dd883207ee9b31c19169adf97c918d561dcb9a5c.
And I verified the fix again. The system can boot successfully.
Regards,
Yuye.
------------------------------------------------------------------
发件人：Olivier Deprez <Olivier.Deprez(a)arm.com>
发送时间：2023年3月23日(星期四) 16:54
收件人：梅建强(禹夜) <meijianqiang.mjq(a)alibaba-inc.com>
主 题：Re: load optee at 36bit address
Hi Yuye, just to acknowledge receipt of this query.
Please accept a slow response as I'm currently away from my office.
I will do a few trials at my end to reproduce and try to help at my best.
For building/running hafnium , did you have to apply a change similar to:
https://review.trustedfirmware.org/c/hafnium/hafnium/+/18510 <https://review.trustedfirmware.org/c/hafnium/hafnium/+/18510 >
Regards,
Olivier.
From: 梅建强(禹夜) <meijianqiang.mjq(a)alibaba-inc.com>
Sent: 22 March 2023 04:50
To: Olivier Deprez <Olivier.Deprez(a)arm.com>; Jens Wiklander <jens.wiklander(a)linaro.org>
Cc: hafnium <hafnium(a)lists.trustedfirmware.org>; op-tee <op-tee(a)lists.trustedfirmware.org>
Subject: load optee at 36bit address
Hi, experts,
We are currently moving optee to a 36bit address to boot with environment
that hafnium runs at sel2 as spmc and optee runs at sel1 as sp.
Now we have moved hafnium to 0x880000000 and run successfully.
Then we tried moving optee to a 36bit address (0x89000000) as well.
Although hafnium and optee were successfully initialized on the primary cpu,
psci_cpu_on does not seem to be entered into when the secondary cpu is started.
The error is as follows:
https://github.com/OP-TEE/optee_os/issues/5895 <https://github.com/OP-TEE/optee_os/issues/5895 >
Is there any difference between the two cases
where hafnium and optee initialize the secondary cpu with different load_address?
The log print shows that the secondary cpu has not entered hafnium.
Could Hafnium be affected by the 36bit address when dealing with psci related transactions?
Regards,
Yuye.
IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.
