Hi, Jens,
We seem to have found the answer following the comment you gave in the issue:
https://github.com/OP-TEE/optee_os/issues/5877 <https://github.com/OP-TEE/optee_os/issues/5877 >
The reason seems to be that the system has more non-secure memory than what OP-TEE is aware of.
When our system memory DRAM size is 32GB, optee has the following configuration for non-secure state memory:
#define DRAM1_BASE 0x880000000UL
#define DRAM1_SIZE 0x780000000UL
On another device, our system memory DRAM size is 64G, optee uses the above configuration for non-secure memory.
And xtest will not run stably.
In addition, I would like to ask you a few questions in order to understand the problem better.
1.
System has more non-secure memory than what OP-TEE is aware of, and optee will not run stably.
Why? What is the root cause of TEEC_ERROR_OUT_OF_MEMORY on the issue?
I'll do some research on it myself, meanwhile I hope you could give us some insights.
2.
When I execute the following command,
for i in {1.. 10}; do ./xtest&; done;
xtest prints out of order,
And the following error is reported:
E/TC:032 002 mobj_ffa_get_by_cookie:356 possible spinlock deadlock reminder 1
E/TC:070 mobj_ffa_unregister_by_cookie:310 possible spinlock deadlock reminder 1
E/TC:004 003 mobj_ffa_get_by_cookie:356 possible spinlock deadlock reminder 1
E/TC:058 001 mobj_ffa_get_by_cookie:356 possible spinlock deadlock reminder 1
Then the system hangs and restarts automatically.
Does optee currently support running multiple TAs in parallel?
Is the maximum number of TAs running in parallel equal to the number of vcpus?
3.
The values of MAX_XLAT_TABLES and CFG_CORE_HEAP_SIZE also seem to affect the stability of the system.
What configurations, such as the number of cores, should I pay attention to if I want to configure appropriate values for these options?
Thanks a lot.
Regards,
Yuye.
------------------------------------------------------------------
发件人：Jens Wiklander <jens.wiklander(a)linaro.org>
发送时间：2023年3月20日(星期一) 15:05
收件人：梅建强(禹夜) <meijianqiang.mjq(a)alibaba-inc.com>
抄 送：Olivier Deprez <Olivier.Deprez(a)arm.com>; hafnium <hafnium(a)lists.trustedfirmware.org>; op-tee <op-tee(a)lists.trustedfirmware.org>
主 题：Re: optee xtest cannot run success stably
Hi Yuye,
Comment below.
On Mon, Mar 20, 2023 at 4:43 AM 梅建强(禹夜)
<meijianqiang.mjq(a)alibaba-inc.com> wrote:
> Hi, experts
>
> Recently, we are testing the stability for running optee xtest with environment that hafnium runs as SPMC and optee runs on SPMC as SP.
> When we reboot the system, xtest failed on some cases with TEEC_ERROR_OUT_OF_MEMORY.
> It seems to be that there is an insufficient memory allocation somewhere in the chain.
> We tried the following:
> Using Single core startup,
> Increased optee MAX_XLAT_TABLES size to 16,
> Increased optee CFG_CORE_HEAP_SIZE to 0x2000000,
> Increasing the size of optee CFG_TEE_RAM_VA_SIZE to 0x4000000,
> Increasing the size of hafnium heap_pages to 8192,
> But nothing seems to be working.
> Can you offer any help or suggestions?
It would help if you could pinpoint the source of the out-of-memory
error. I guess it happens somewhere during mobj_ffa_get_by_cookie(),
where especially thread_spmc_populate_mobj_from_rx() is interesting.
It could also be worth setting CFG_CORE_DUMP_OOM=y, it's easy to
enable but I'm afraid it's more of a long shot.
Cheers,
Jens
> Some other configuration for optee is attached in the issue:
> https://github.com/OP-TEE/optee_os/issues/5893 <https://github.com/OP-TEE/optee_os/issues/5893 >
>
> Regards,
> Yuye.
>
Hi, experts
Recently, we are testing the stability for running optee xtest with environment that hafnium runs as SPMC and optee runs on SPMC as SP.
When we reboot the system, xtest failed on some cases with TEEC_ERROR_OUT_OF_MEMORY.
It seems to be that there is an insufficient memory allocation somewhere in the chain.
We tried the following:
Using Single core startup,
Increased optee MAX_XLAT_TABLES size to 16,
Increased optee CFG_CORE_HEAP_SIZE to 0x2000000,
Increasing the size of optee CFG_TEE_RAM_VA_SIZE to 0x4000000,
Increasing the size of hafnium heap_pages to 8192,
But nothing seems to be working.
Can you offer any help or suggestions?
Some other configuration for optee is attached in the issue:
https://github.com/OP-TEE/optee_os/issues/5893 <https://github.com/OP-TEE/optee_os/issues/5893 >
Regards,
Yuye.
Hi,
With the introduction of FFA_CONSOLE_LOG ABI [1], we are intending to replace and remove support for HF_DEBUG_LOG.
This proposal is in review in the following stages:
1) Remove the dependency of hftest VMs on HF_DEBUG_LOG and move to FFA_CONSOLE_LOG [2]
2) Remove the support for HF_DEBUG_LOG (i.e. api_debug_log) from hafnium project. [3]
The adoption of FFA_CONSOLE_LOG will allow us to make use of its ability to log multiple characters at a time, as opposed to HF_DEBUG_LOG which writes one character at a time.
This improvement will be enabled in a future patch. Also, should [3] be adopted, we will make accompanying changes to tf-a-tests Cactus-based tests.
We want to know if there are any concerns about removing support for HF_DEBUG_LOG at this time as we realize other downstream SPs may rely on its support.
Thank you,
Kathleen Capella
[1] feat(console_log): add FFA_CONSOLE_LOG ABI https://review.trustedfirmware.org/c/hafnium/hafnium/+/15334
[2] feat(ffa_console_log): replace hf_debug_log https://review.trustedfirmware.org/c/hafnium/hafnium/+/19513
[3] refactor: remove support for HF_DEBUG_LOG https://review.trustedfirmware.org/c/hafnium/hafnium/+/19681
Hi, Olivier
I have forgotten one thing, hafnium will invalidate the data cache for the whole image.
adrp x0, ORIGIN_ADDRESS
adrp x1, image_end
sub x1, x1, x0
bl arch_cache_data_invalidate_range
I solved this problem by using an address-fixed variable outside the data section instead of the global variabal malloced in data section.
In addition, I would like to ask that
if multiple optees run on the Hafnium, how to switch the vCPUs of different optees?
If one of the optees is expected to be boot under the host OS (post-boot load), how to switch the vCPUs?
Thanks for the support.
Regards,
Yuye.
------------------------------------------------------------------
发件人：梅建强(禹夜) <meijianqiang.mjq(a)alibaba-inc.com>
发送时间：2023年3月11日(星期六) 21:36
收件人：Olivier Deprez <Olivier.Deprez(a)arm.com>
抄 送：Jens Wiklander <jens.wiklander(a)linaro.org>; hafnium(a)lists.trustedfirmware.org <hafnium(a)lists.trustedfirmware.org>
主 题：Re: Multi-OPTEE run with Hafnium
Hi, Olivier,
The framework message definition that I mentioned is refered in the FF-A Specification.
I hope I didn't misinterpret it.
The work flow is interpreted as follows:
When the system run into the host OS,
I want implemented the post-boot load optee as mentioned in the email with Hafnium as SPMC
https://review.trustedfirmware.org/c/TF-A/trusted-firmware-a/+/18635 <https://review.trustedfirmware.org/c/TF-A/trusted-firmware-a/+/18635 >
However, when the intialization about hafnium and optee has been finished,
SMC_RET8 instruction failed reback from spmd to optee driver.
I suspect that there may be some special handling of context needed here.
Can you provide some suggestions or help?
Regards,
Yuye.
------------------------------------------------------------------
发件人：Olivier Deprez <Olivier.Deprez(a)arm.com>
发送时间：2023年3月7日(星期二) 16:55
收件人：梅建强(禹夜) <meijianqiang.mjq(a)alibaba-inc.com>
抄 送：Jens Wiklander <jens.wiklander(a)linaro.org>; hafnium(a)lists.trustedfirmware.org <hafnium(a)lists.trustedfirmware.org>
主 题：Re: Multi-OPTEE run with Hafnium
Hi Yuye,
I sense those questions are related to downstream changes of yours, and I may not be able to provide accurate answers without seeing the code.
Please clarify if the questions are related to the upstream version of Hafnium.
See additional comments below [OD].
Regards,
Olivier.
From: 梅建强(禹夜) <meijianqiang.mjq(a)alibaba-inc.com>
Sent: 06 March 2023 14:15
To: Olivier Deprez <Olivier.Deprez(a)arm.com>
Cc: Jens Wiklander <jens.wiklander(a)linaro.org>
Subject: Re: Multi-OPTEE run with Hafnium
Hi, Olivier,
On this question, I should add something based on the results of my tests.
When I enter Hafnium through framwork message, the value of the global variable remains the same.
[OD] please explain what is a framework message.
When I re-enter the Hafnium image_entry via spmd_init, the global variable is restored to its original value.
[OD] I'm not sure about the flow here, it depends about where the variable is declared and whether you have local changes to the hafnium boot flows.
Do you know why?
Regards,
Yuye.
------------------------------------------------------------------
发件人：梅建强(禹夜) <meijianqiang.mjq(a)alibaba-inc.com>
发送时间：2023年3月6日(星期一) 17:00
收件人：Olivier Deprez <olivier.deprez(a)arm.com>
抄 送：Jens Wiklander <jens.wiklander(a)linaro.org>
主 题：Multi-OPTEE run with Hafnium
Hi, Olivier,
I want to ask a question about variable modification.
Before this, let me explain the background about the question.
Now, I have run into host OS with Hafnium as spmc and OPTEE as SP.
Then the optee driver will send a framwork message to Hafnium to inform it that whether the current state of the host OS is booting or running,
[OD] What is a 'framework message' in this context? Is this a custom direct request from driver to optee?
so that Hafnium could know which vm node(SP) it should load and init at current state.
[OD] Can you please explain why this is required?
This current state is represented by a global variable.
[OD] do you mean a variable in Hafnium's BSS space?
Will the global variable change when message flow exit Hafnium and enter it again?
[OD] This depends in which section the variable is declared (see above).
Regards,
Yuye.
Hi, Olivier,
The framework message definition that I mentioned is refered in the FF-A Specification.
I hope I didn't misinterpret it.
The work flow is interpreted as follows:
When the system run into the host OS,
I want implemented the post-boot load optee as mentioned in the email with Hafnium as SPMC
https://review.trustedfirmware.org/c/TF-A/trusted-firmware-a/+/18635 <https://review.trustedfirmware.org/c/TF-A/trusted-firmware-a/+/18635 >
However, when the intialization about hafnium and optee has been finished,
SMC_RET8 instruction failed reback from spmd to optee driver.
I suspect that there may be some special handling of context needed here.
Can you provide some suggestions or help?
Regards,
Yuye.
------------------------------------------------------------------
发件人：Olivier Deprez <Olivier.Deprez(a)arm.com>
发送时间：2023年3月7日(星期二) 16:55
收件人：梅建强(禹夜) <meijianqiang.mjq(a)alibaba-inc.com>
抄 送：Jens Wiklander <jens.wiklander(a)linaro.org>; hafnium(a)lists.trustedfirmware.org <hafnium(a)lists.trustedfirmware.org>
主 题：Re: Multi-OPTEE run with Hafnium
Hi Yuye,
I sense those questions are related to downstream changes of yours, and I may not be able to provide accurate answers without seeing the code.
Please clarify if the questions are related to the upstream version of Hafnium.
See additional comments below [OD].
Regards,
Olivier.
From: 梅建强(禹夜) <meijianqiang.mjq(a)alibaba-inc.com>
Sent: 06 March 2023 14:15
To: Olivier Deprez <Olivier.Deprez(a)arm.com>
Cc: Jens Wiklander <jens.wiklander(a)linaro.org>
Subject: Re: Multi-OPTEE run with Hafnium
Hi, Olivier,
On this question, I should add something based on the results of my tests.
When I enter Hafnium through framwork message, the value of the global variable remains the same.
[OD] please explain what is a framework message.
When I re-enter the Hafnium image_entry via spmd_init, the global variable is restored to its original value.
[OD] I'm not sure about the flow here, it depends about where the variable is declared and whether you have local changes to the hafnium boot flows.
Do you know why?
Regards,
Yuye.
------------------------------------------------------------------
发件人：梅建强(禹夜) <meijianqiang.mjq(a)alibaba-inc.com>
发送时间：2023年3月6日(星期一) 17:00
收件人：Olivier Deprez <olivier.deprez(a)arm.com>
抄 送：Jens Wiklander <jens.wiklander(a)linaro.org>
主 题：Multi-OPTEE run with Hafnium
Hi, Olivier,
I want to ask a question about variable modification.
Before this, let me explain the background about the question.
Now, I have run into host OS with Hafnium as spmc and OPTEE as SP.
Then the optee driver will send a framwork message to Hafnium to inform it that whether the current state of the host OS is booting or running,
[OD] What is a 'framework message' in this context? Is this a custom direct request from driver to optee?
so that Hafnium could know which vm node(SP) it should load and init at current state.
[OD] Can you please explain why this is required?
This current state is represented by a global variable.
[OD] do you mean a variable in Hafnium's BSS space?
Will the global variable change when message flow exit Hafnium and enter it again?
[OD] This depends in which section the variable is declared (see above).
Regards,
Yuye.
Hi, Olivier,
For the question in part 3, I think I didn't express clearly.
>Does Hafnium support the configuration of the location of running memory such as text, data, stacks, and heap? If yes, how?
During Hafnium initialization, I can see the following memory spaces about text, rodata, data and stacks.
INFO: Initializing Hafnium (SPMC)
INFO: text: 0xff200000 - 0xff225000
INFO: rodata: 0xff225000 - 0xff22e000
INFO: data: 0xff22e000 - 0xffbcc000
INFO: stacks: 0xffbd0000 - 0xffcd0000
Are the sizes and start_address of these memory spaces configurable? If yes, how?
In addition, I see the configuration of memory node in spmc_optee_sp_manifest.dts, and I modify it as follows:
memory@0xFF200000 {
device_type = "memory";
reg = <0x0 0xff200000 0x00a00000>,
<0x8 0x80000000 0x10000000>;/* Trusted DRAM */
};
Is the modification reasonable?
Does this node represent all the memory managed by Hafnium?
Should the region specified by the memory node include the optee memory spaces?
Regards,
Yuye.
------------------------------------------------------------------
发件人：Olivier Deprez <Olivier.Deprez(a)arm.com>
发送时间：2023年2月27日(星期一) 22:13
收件人：梅建强(禹夜) <meijianqiang.mjq(a)alibaba-inc.com>
抄 送：Jens Wiklander <jens.wiklander(a)linaro.org>; hafnium <hafnium(a)lists.trustedfirmware.org>
主 题：Re: [Hafnium] Re: Re: Re: xtest 1034
Hi Yuye,
See comments inline [OD]
Regards,
Olivier.
From: 梅建强(禹夜) <meijianqiang.mjq(a)alibaba-inc.com>
Sent: 27 February 2023 12:52
To: Olivier Deprez <Olivier.Deprez(a)arm.com>
Cc: Jens Wiklander <jens.wiklander(a)linaro.org>; hafnium <hafnium(a)lists.trustedfirmware.org>
Subject: [Hafnium] Re: Re: Re: xtest 1034
Hi, Olivier,
Sorry for ignoring the tips in your email.
I guess the question in part 1 is no longer the trouble.
[OD]No worries.
Regards,
Yuye.
------------------------------------------------------------------
发件人：梅建强(禹夜) <meijianqiang.mjq(a)alibaba-inc.com>
发送时间：2023年2月27日(星期一) 19:18
收件人：Olivier Deprez <olivier.deprez(a)arm.com>
抄 送：Jens Wiklander <jens.wiklander(a)linaro.org>; hafnium <hafnium(a)lists.trustedfirmware.org>
主 题：[Hafnium] Re: Re: xtest 1034
Hi, Olivier,
1.
We solved the problem by update our Hafnium codebase on rebase of commit ca03054ba6a351534b93e6d64c12e671578eb340.
The origin Hafnium codebase is on rebase of commit dd883207ee9b31c19169adf97c918d561dcb9a5c.
OPTEE codebase stays with version 3.20.0.
Which commit can solve this problem is still under investigation.
I guess the reason may relate to that Hafnium has updated the implementation of FF-A mechanism in new commit.
If you have already known the reason, please tell me.
2.
In addition, although some cases run successfully, there are still many error messages generated during their running process,
such as the following cases:
* regression_1034 Test loading a large TA
E/TC:011 0 mobj_ffa_get_by_cookie:384 Populating mobj from rx buffer, cookie 0x1
E/TC:011 0 mobj_ffa_get_by_cookie:384 Populating mobj from rx buffer, cookie 0x3
E/TC:??? 0 mobj_ffa_get_by_cookie:423 Failed to get cookie 0x3 internal_offs 0
E/LD: init_elf:486 sys_open_ta_bin(25497083-a58a-4fc5-8a72-1ad7b69b8562)
E/TC:??? 0 ldelf_init_with_ldelf:131 ldelf failed with res: 0xffff000c
TEEC_ERROR_OUT_OF_MEMORY - ignored
regression_1034 OK
[OD] I noticed this some time ago but I believe this might be related to some integration problem (e.g. lacking memory available to linux?)
Moreover the test result being ignored suggests that this failure may happen but is eventually harmless from the xtest suite perspective.
Jens & Jerome might know better.
Note that I'm seeing this test passing with sw stack integrations targeting two different models (qemu and TC).
* regression_1022 Test dlopen()/dlsym()/dlclose() API
E/TC:011 0 mobj_ffa_get_by_cookie:384 Populating mobj from rx buffer, cookie 0x1
E/TC:011 0 mobj_ffa_get_by_cookie:384 Populating mobj from rx buffer, cookie 0x3
E/TC:011 0 mobj_ffa_get_by_cookie:384 Populating mobj from rx buffer, cookie 0x3
E/TC:011 0 mobj_ffa_get_by_cookie:384 Populating mobj from rx buffer, cookie 0x1
E/TC:011 0 mobj_ffa_get_by_cookie:384 Populating mobj from rx buffer, cookie 0x3
E/TC:??? 0
E/TC:??? 0 TA panicked with code 0x0
E/LD: Status of TA 5b9e0e40-2636-11e1-ad9e-0002a5d5c51b
E/LD: arch: aarch64
E/LD: region 0: va 0x40006000 pa 0xf0404000 size 0x002000 flags rw-s (ldelf)
E/LD: region 1: va 0x40008000 pa 0xf0406000 size 0x011000 flags r-xs (ldelf)
E/LD: region 2: va 0x40019000 pa 0xf0417000 size 0x001000 flags rw-s (ldelf)
E/LD: region 3: va 0x4001a000 pa 0xf0418000 size 0x004000 flags rw-s (ldelf)
E/LD: region 4: va 0x4001e000 pa 0xf041c000 size 0x001000 flags r--s
E/LD: region 5: va 0x4001f000 pa 0xf052e000 size 0x003000 flags rw-s (stack)
E/LD: region 6: va 0x40038000 pa 0x00000000 size 0x00f000 flags r-xs [2]
E/LD: region 7: va 0x40047000 pa 0x0000e000 size 0x002000 flags rw-s [2]
E/LD: region 8: va 0x40054000 pa 0x00000000 size 0x00f000 flags r-xs [1]
E/LD: region 9: va 0x40063000 pa 0x0000e000 size 0x002000 flags rw-s [1]
E/LD: region 10: va 0x4008b000 pa 0x00001000 size 0x02a000 flags r-xs [0]
E/LD: region 11: va 0x400b5000 pa 0x0002b000 size 0x0e7000 flags rw-s [0]
E/LD: [0] 5b9e0e40-2636-11e1-ad9e-0002a5d5c51b @ 0x4008b000
E/LD: [1] ffd2bded-ab7d-4988-95ee-e4962fff7154 @ 0x40054000
E/LD: [2] b3091a65-9751-4784-abf7-0298a7cc35ba @ 0x40038000
E/LD: Call stack:
E/LD: 0x4009e5a8
E/LD: 0x400a09b8
E/LD: 0x4009ed0c
regression_1022 OK
The E/TC message generated by mobj_ffa_get_by_cookie appears in almost all cases.
Is this related to FFA_MEM_FRAG_RX/TX, or something else?
And what is related to the E/LD message?
I am currently studying the FFA specification to try to explain these problems.
[OD]I have the same logs with qemu and TC platforms. I did not consider the TA panic as an issue provided the regression_1022 test result reports as 'OK'.
Actually, that may be an expected behaviour from the test? Again Jens, Jerome, feel free to comment.
And some related questions raised in the other email(subject: Hafnium Dynamic Shared Memory).
[OD]I will come back to this other email thread, but I don't believe above 'issues' are related.
You can tell me if it's convenient for you.
3.
Could memory managed by Hafnium be configured in any other way except by spmc manifest?
Does Hafnium support the configuration of the location of running memory such as text, data, stacks, and heap? If yes, how?
[OD] I'm not sure I understand the question, perhaps can you tell in more details what you want to achieve?
Thanks you for your help.
Regards,
Yuye.
------------------------------------------------------------------
发件人：Olivier Deprez via Hafnium <hafnium(a)lists.trustedfirmware.org>
发送时间：2023年2月27日(星期一) 17:19
收件人：梅建强(禹夜) <meijianqiang.mjq(a)alibaba-inc.com>
抄 送：hafnium <hafnium(a)lists.trustedfirmware.org>
主 题：[Hafnium] Re: xtest 1034
Hi Yuye,
Can you make sure to use hafnium tip of master (if that's not the case)?
We confirmed this specific test passes on models from this commit onwards:
https://git.trustedfirmware.org/hafnium/hafnium.git/commit/?id=fdd29277caf2… <https://git.trustedfirmware.org/hafnium/hafnium.git/commit/?id=fdd29277caf2… >
We may provide further guidance if the issue persists after rebasing.
Thanks, Olivier.
________________________________
From: 梅建强(禹夜) <meijianqiang.mjq(a)alibaba-inc.com>
Sent: 26 February 2023 06:49
To: hafnium <hafnium(a)lists.trustedfirmware.org>; Olivier Deprez <Olivier.Deprez(a)arm.com>; Jens Wiklander <jens.wiklander(a)linaro.org>
Subject: xtest 1034
Hi, experts,
For dynamic shared memory usage, I am not very clear.
So we need to consult you about the related question.
I failed to run xtest 1034 when I used Hafnium as SPMC and OPTEE as SP.
Here are the debug logs：
#./xtest 1034&
[2] 10130
Test ID: 1034
Run test suite with level=0
TEE test application started over default TEE instance
######################################################
#
# regression
#
######################################################
regression_1034 Test loading a large TA
WARNING: Atf_Debug(55): 0x8400006f 0x8001 0x0 0x80000000 0x0 0x0 0x0 0x0
VERBOSE: Hafnium_Debug ffa_handler func:0x8400006f
D/TC:055 0 mobj_ffa_get_by_cookie:381 cookie 0 resurrecting
D/TC:??? 0 tee_ta_init_pseudo_ta_session:296 Lookup pseudo TA 25497083-a58a-4fc5-8a72-1ad7b69b8562
D/TC:??? 0 ldelf_load_ldelf:96 ldelf load address 0x40008000
D/LD: ldelf:134 Loading TS 25497083-a58a-4fc5-8a72-1ad7b69b8562
F/TC:??? 0 trace_syscall:151 syscall Add constant time memcmp_ct function #3<https://github.com/OP-TEE/optee_os/pull/3> <https://github.com/OP-TEE/optee_os/pull/3> > (syscall_get_property)
F/TC:??? 0 trace_syscall:151 syscall GitHub usage documentation #5<https://github.com/OP-TEE/optee_os/pull/5> <https://github.com/OP-TEE/optee_os/pull/5> > (syscall_open_ta_session)
#D/TC:??? 0 ldelf_syscall_open_bin:143 Lookup user TA ELF 25497083-a58a-4fc5-8a72-1ad7b69b8562 (early TA)
D/TC:??? 0 ldelf_syscall_open_bin:146 res=0xffff0008
D/TC:??? 0 ldelf_syscall_open_bin:143 Lookup user TA ELF 25497083-a58a-4fc5-8a72-1ad7b69b8562 (Secure Storage TA)
VERBOSE: Hafnium_Debug ffa_handler func:0x84000070
WARNING: Atf_Debug(55): 0x84000070 0x80010000 0x0 0x0 0x1 0x0 0x0 0x0
WARNING: Atf_Debug(60): 0x84000073 0x50 0x50 0x0 0x0 0x0 0x0 0x0
VERBOSE: Hafnium_Debug ffa_handler func:0x84000073
VERBOSE: Marked sending complete.
Current share states:
SHARE 0x0 (from VM 0x0, attributes 0x2f, flags 0x8, tag 0, to 1 recipients [VM 0x8001: 0x6 (offset 48)]): fully sent with 1 fragments, 1 retrieved, sender's original mode: 0x7
SHARE 0x1 (from VM 0x0, attributes 0x2f, flags 0x8, tag 0, to 1 recipients [VM 0x8001: 0x6 (offset 48)]): fully sent with 1 fragments, 0 retrieved, sender's original mode: 0x7
WARNING: Atf_Debug(60): 0x84000061 0x0 0x1 0x0 0x0 0x0 0x0 0x0
WARNING: Atf_Debug(55): 0x8400006f 0x8001 0x0 0x80000001 0x0 0x0 0x0 0x0
VERBOSE: Hafnium_Debug ffa_handler func:0x8400006f
E/TC:055 0 mobj_ffa_get_by_cookie:384 Populating mobj from rx buffer, cookie 0x1
VERBOSE: Hafnium_Debug ffa_handler func:0x84000074
Current share states:
SHARE 0x0 (from VM 0x0, attributes 0x2f, flags 0x8, tag 0, to 1 recipients [VM 0x8001: 0x6 (offset 48)]): fully sent with 1 fragments, 1 retrieved, sender's original mode: 0x7
SHARE 0x1 (from VM 0x0, attributes 0x2f, flags 0x8, tag 0, to 1 recipients [VM 0x8001: 0x6 (offset 48)]): fully sent with 1 fragments, 0 retrieved, sender's original mode: 0x7
Current share states:
SHARE 0x0 (from VM 0x0, attributes 0x2f, flags 0x8, tag 0, to 1 recipients [VM 0x8001: 0x6 (offset 48)]): fully sent with 1 fragments, 1 retrieved, sender's original mode: 0x7
SHARE 0x1 (from VM 0x0, attributes 0x2f, flags 0x8, tag 0, to 1 recipients [VM 0x8001: 0x6 (offset 48)]): fully sent with 1 fragments, 1 retrieved, sender's original mode: 0x7
VERBOSE: Hafnium_Debug ffa_handler func:0x84000065
VERBOSE: Hafnium_Debug ffa_handler func:0x84000070
WARNING: Atf_Debug(55): 0x84000070 0x80010000 0x0 0x0 0x1 0x0 0x0 0x0
WARNING: Atf_Debug(55): 0x8400006f 0x8001 0x0 0x80000001 0x0 0x0 0x0 0x0
VERBOSE: Hafnium_Debug ffa_handler func:0x8400006f
VERBOSE: Hafnium_Debug ffa_handler func:0x84000070
WARNING: Atf_Debug(55): 0x84000070 0x80010000 0x0 0x0 0x1 0x0 0x0 0x0
WARNING: Atf_Debug(55): 0x8400006f 0x8001 0x0 0x80000001 0x0 0x0 0x0 0x0
VERBOSE: Hafnium_Debug ffa_handler func:0x8400006f
VERBOSE: Hafnium_Debug ffa_handler func:0x84000070
WARNING: Atf_Debug(55): 0x84000070 0x80010000 0x0 0x0 0x1 0x0 0x0 0x0
WARNING: Atf_Debug(55): 0x8400006f 0x8001 0x0 0x80000001 0x0 0x0 0x0 0x0
VERBOSE: Hafnium_Debug ffa_handler func:0x8400006f
VERBOSE: Hafnium_Debug ffa_handler func:0x84000070
WARNING: Atf_Debug(55): 0x84000070 0x80010000 0x0 0x0 0x1 0x0 0x0 0x0
WARNING: Atf_Debug(55): 0x8400006f 0x8001 0x0 0x80000001 0x0 0x0 0x0 0x0
VERBOSE: Hafnium_Debug ffa_handler func:0x8400006f
VERBOSE: Hafnium_Debug ffa_handler func:0x84000070
WARNING: Atf_Debug(55): 0x84000070 0x80010000 0x0 0x0 0x1 0x0 0x0 0x0
WARNING: Atf_Debug(55): 0x8400006f 0x8001 0x0 0x80000001 0x0 0x0 0x0 0x0
VERBOSE: Hafnium_Debug ffa_handler func:0x8400006f
VERBOSE: Hafnium_Debug ffa_handler func:0x84000070
WARNING: Atf_Debug(55): 0x84000070 0x80010000 0x0 0x0 0x1 0x0 0x0 0x0
WARNING: Atf_Debug(55): 0x8400006f 0x8001 0x0 0x80000001 0x0 0x0 0x0 0x0
VERBOSE: Hafnium_Debug ffa_handler func:0x8400006f
VERBOSE: Hafnium_Debug ffa_handler func:0x84000070
WARNING: Atf_Debug(55): 0x84000070 0x80010000 0x0 0x0 0x1 0x0 0x0 0x0
WARNING: Atf_Debug(55): 0x8400006f 0x8001 0x0 0x80000001 0x0 0x0 0x0 0x0
VERBOSE: Hafnium_Debug ffa_handler func:0x8400006f
VERBOSE: Hafnium_Debug ffa_handler func:0x84000070
WARNING: Atf_Debug(55): 0x84000070 0x80010000 0x0 0x0 0x1 0x0 0x0 0x0
WARNING: Atf_Debug(55): 0x8400006f 0x8001 0x0 0x80000001 0x0 0x0 0x0 0x0
VERBOSE: Hafnium_Debug ffa_handler func:0x8400006f
VERBOSE: Hafnium_Debug ffa_handler func:0x84000070
WARNING: Atf_Debug(55): 0x84000070 0x80010000 0x0 0x0 0x1 0x0 0x0 0x0
WARNING: Atf_Debug(55): 0x8400006f 0x8001 0x0 0x80000001 0x0 0x0 0x0 0x0
VERBOSE: Hafnium_Debug ffa_handler func:0x8400006f
VERBOSE: Hafnium_Debug ffa_handler func:0x84000070
WARNING: Atf_Debug(55): 0x84000070 0x80010000 0x0 0x0 0x1 0x0 0x0 0x0
WARNING: Atf_Debug(55): 0x8400006f 0x8001 0x0 0x80000001 0x0 0x0 0x0 0x0
VERBOSE: Hafnium_Debug ffa_handler func:0x8400006f
VERBOSE: Hafnium_Debug ffa_handler func:0x84000070
WARNING: Atf_Debug(55): 0x84000070 0x80010000 0x0 0x0 0x1 0x0 0x0 0x0
WARNING: Atf_Debug(55): 0x8400006f 0x8001 0x0 0x80000001 0x0 0x0 0x0 0x0
VERBOSE: Hafnium_Debug ffa_handler func:0x8400006f
VERBOSE: Hafnium_Debug ffa_handler func:0x84000070
WARNING: Atf_Debug(55): 0x84000070 0x80010000 0x0 0x0 0x1 0x0 0x0 0x0
WARNING: Atf_Debug(55): 0x8400006f 0x8001 0x0 0x80000001 0x0 0x0 0x0 0x0
VERBOSE: Hafnium_Debug ffa_handler func:0x8400006f
VERBOSE: Hafnium_Debug ffa_handler func:0x84000070
WARNING: Atf_Debug(55): 0x84000070 0x80010000 0x0 0x0 0x1 0x0 0x0 0x0
WARNING: Atf_Debug(55): 0x8400006f 0x8001 0x0 0x80000001 0x0 0x0 0x0 0x0
VERBOSE: Hafnium_Debug ffa_handler func:0x8400006f
VERBOSE: Hafnium_Debug ffa_handler func:0x84000070
WARNING: Atf_Debug(55): 0x84000070 0x80010000 0x0 0x0 0x1 0x0 0x0 0x0
WARNING: Atf_Debug(55): 0x8400006f 0x8001 0x0 0x80000001 0x0 0x0 0x0 0x0
VERBOSE: Hafnium_Debug ffa_handler func:0x8400006f
VERBOSE: Hafnium_Debug ffa_handler func:0x84000070
WARNING: Atf_Debug(55): 0x84000070 0x80010000 0x0 0x0 0x1 0x0 0x0 0x0
WARNING: Atf_Debug(55): 0x8400006f 0x8001 0x0 0x80000001 0x0 0x0 0x0 0x0
VERBOSE: Hafnium_Debug ffa_handler func:0x8400006f
VERBOSE: Hafnium_Debug ffa_handler func:0x84000070
WARNING: Atf_Debug(55): 0x84000070 0x80010000 0x0 0x0 0x1 0x0 0x0 0x0
WARNING: Atf_Debug(55): 0x8400006f 0x8001 0x0 0x80000001 0x0 0x0 0x0 0x0
VERBOSE: Hafnium_Debug ffa_handler func:0x8400006f
VERBOSE: Hafnium_Debug ffa_handler func:0x84000070
WARNING: Atf_Debug(55): 0x84000070 0x80010000 0x0 0x0 0x1 0x0 0x0 0x0
WARNING: Atf_Debug(55): 0x8400006f 0x8001 0x0 0x80000001 0x0 0x0 0x0 0x0
VERBOSE: Hafnium_Debug ffa_handler func:0x8400006f
D/TC:??? 0 ldelf_syscall_open_bin:146 res=0xffff0008
D/TC:??? 0 ldelf_syscall_open_bin:143 Lookup user TA ELF 25497083-a58a-4fc5-8a72-1ad7b69b8562 (REE)
VERBOSE: Hafnium_Debug ffa_handler func:0x84000070
WARNING: Atf_Debug(55): 0x84000070 0x80010000 0x0 0x0 0x1 0x0 0x0 0x0
WARNING: Atf_Debug(55): 0x8400006f 0x8001 0x0 0x80000001 0x0 0x0 0x0 0x0
VERBOSE: Hafnium_Debug ffa_handler func:0x8400006f
VERBOSE: Hafnium_Debug ffa_handler func:0x84000070
WARNING: Atf_Debug(55): 0x84000070 0x80010000 0x0 0x0 0x1 0x0 0x0 0x0
WARNING: Atf_Debug(58): 0x84000073 0x1190 0x1000 0x0 0x0 0x0 0x0 0x0
VERBOSE: Hafnium_Debug ffa_handler func:0x84000073
Current share states:
SHARE 0x0 (from VM 0x0, attributes 0x2f, flags 0x8, tag 0, to 1 recipients [VM 0x8001: 0x6 (offset 48)]): fully sent with 1 fragments, 1 retrieved, sender's original mode: 0x7
SHARE 0x1 (from VM 0x0, attributes 0x2f, flags 0x8, tag 0, to 1 recipients [VM 0x8001: 0x6 (offset 48)]): fully sent with 1 fragments, 1 retrieved, sender's original mode: 0x7
SHARE 0x2 (from VM 0x0, attributes 0x2f, flags 0x8, tag 0, to 1 recipients [VM 0x8001: 0x6 (offset 48)]): partially sent with 1 fragments, 0 retrieved, sender's original mode: 0x7
WARNING: Atf_Debug(58): 0x8400007a 0x2 0x0 0x1000 0x0 0x0 0x0 0x0
WARNING: Atf_Debug(58): 0x8400007b 0x2 0x0 0x190 0x0 0x0 0x0 0x0
VERBOSE: Hafnium_Debug ffa_handler func:0x8400007b
VERBOSE: Hafnium_Debug fragment_length:0x190
VERBOSE: Hafnium_Debug fragment_copy:00000000ff30e000, from_msg:00000008bdd48000
ERROR: Data abort: pc=0xff21a688, esr=0x96000006, ec=0x25, far=0x9c
Panic: EL2 exception
the error occured when Hafnium run the code:
api_ffa_mem_frag_tx
memcpy_s(fragment_copy, MM_PPOOL_ENTRY_SIZE, from_msg, fragment_length);
It seems that I did not add the page table at 0x00000000ff30e000 or 0x00000008bdd48000,
but I am not quite clear about what fragment_copy and from_msg mean.
Can someone help me see what the problem is?
Regards,
Yuye.
--
Hafnium mailing list -- hafnium(a)lists.trustedfirmware.org
To unsubscribe send an email to hafnium-leave(a)lists.trustedfirmware.org
Hi Olivier,
Fix from https://github.com/OP-TEE/optee_os/issues/5803#issuecomment-1436084763 <https://github.com/OP-TEE/optee_os/issues/5803#issuecomment-1436084763 > has been confirmed in the comments.
Regards,
Yuye.
------------------------------------------------------------------
发件人：Olivier Deprez <Olivier.Deprez(a)arm.com>
发送时间：2023年2月28日(星期二) 01:57
收件人：hafnium <hafnium(a)lists.trustedfirmware.org>; 梅建强(禹夜) <meijianqiang.mjq(a)alibaba-inc.com>
抄 送：高海源(码源) <haiyuan.ghy(a)alibaba-inc.com>; 王一蒙(北见) <wym389994(a)alibaba-inc.com>
主 题：Re: [Hafnium] hafnium page table configuration
Hi Yuye,
See comments inline [OD].
Regards,
Olivier.
From: 梅建强(禹夜) <meijianqiang.mjq(a)alibaba-inc.com>
Sent: 22 February 2023 04:17
To: Olivier Deprez <Olivier.Deprez(a)arm.com>; hafnium <hafnium(a)lists.trustedfirmware.org>
Cc: 高海源(码源) <haiyuan.ghy(a)alibaba-inc.com>; 王一蒙(北见) <wym389994(a)alibaba-inc.com>
Subject: 回复：[Hafnium] hafnium page table configuration
Hi, Olivier,
Sorry for the delay in replying to this query.
As I've seen in the documentation, in the case of hafnium implemented as S-EL2,
NWd uses shared memory to communicate with SWd and there are two steps to register the page table.
Please correct me if there are any errors in my description.
Firstly, Linux sends FFA_MEM_SHARE to SPMC (Hafnium), which completes memory mapping for S-EL1's stage-2 transition.
Secondly, Linux sends FFA_MSG_SEND_DIRECT_REQ to OPTEE, and OPTEE then retrieves the IPA space from SPMC according to the cookie received
and then completes the memory mapping for S-EL1's stage-1 transition.
[OD] That's quite correct, with the subtle difference that the region is mapped in SP's S2 page tables upon the receiver SP emitting the memory retrieve request.
The SP maps the region in the S1 page tables after receiving the memory retrieve response from the SPMC.
Our solution to the problem is: After the first step be completed, we add the invalidation of TLB,
which seems to greatly reduce the probability of problem occurrence when testing the optee_examples.
The root cause still needs to be further located.
[OD] There is a probable miss with invalidating the S2 TLB entries for NS IPA space.
Interestingly, this issue is not observed with models, but likely to happen on real silicon.
Can you try the suggested temp. fix from https://github.com/OP-TEE/optee_os/issues/5803#issuecomment-1436084763 <https://github.com/OP-TEE/optee_os/issues/5803#issuecomment-1436084763 > ?
(while reverting your own TLB invalidation fix. ?)
I'm working on a cleaner fix, and prioritize if this issue is confirmed at your end.
It should be noted that our OPTEE version is based on 3.19-rc,
and Hafnium version is based on the commit:
https://git.trustedfirmware.org/hafnium/hafnium.git/commit/?id=dd883207ee9b… <https://git.trustedfirmware.org/hafnium/hafnium.git/commit/?id=dd883207ee9b… >
Anyway, thanks for the support.
Regards,
Yuye.
------------------------------------------------------------------
发件人：Olivier Deprez <Olivier.Deprez(a)arm.com>
发送时间：2023年2月11日(星期六) 00:56
收件人：hafnium <hafnium(a)lists.trustedfirmware.org>; 梅建强(禹夜) <meijianqiang.mjq(a)alibaba-inc.com>
主 题：Re: [Hafnium] hafnium page table configuration
Hi Yuye,
Quick feedback, we have a test case testing SP to SP mem sharing with a large physical address (for a NS memory region), so I expect this is covered:
https://git.trustedfirmware.org/TF-A/tf-a-tests.git/tree/tftf/tests/runtime… <https://git.trustedfirmware.org/TF-A/tf-a-tests.git/tree/tftf/tests/runtime… >
https://git.trustedfirmware.org/TF-A/tf-a-tests.git/tree/spm/cactus/cactus_… <https://git.trustedfirmware.org/TF-A/tf-a-tests.git/tree/spm/cactus/cactus_… >
https://git.trustedfirmware.org/TF-A/tf-a-tests.git/tree/spm/cactus/plat/ar… <https://git.trustedfirmware.org/TF-A/tf-a-tests.git/tree/spm/cactus/plat/ar… >
There may certainly be other reasons, but before investigating further, is this still an issue at your end?
Regards,
Olivier.
From: 梅建强(禹夜) via Hafnium <hafnium(a)lists.trustedfirmware.org>
Sent: 06 February 2023 07:41
To: hafnium <hafnium(a)lists.trustedfirmware.org>
Subject: [Hafnium] hafnium page table configuration
Hi,
At present, I suspect that this may be a hafnium problem,
secure_storage CA/TA may use over 34G address space before communication.
May I ask, does hafnium currently support the configuration of the page table of 0x8 80000000-xxxxxxxxx address range?
If yes, how to configure it?
The error log is as follows:
Current share states: SHARE 0x0 (from VM 0x0, attributes 0x2f, flags 0x8, tag 0, to 1 recipients [VM 0x8001: 0x6 (offset 48)]): fully sent with 1 fragments, 1 retrieved, sender's original mode: 0x7 SHARE 0x1 (from VM 0x0, attributes 0x2f, flags 0x8, tag 0, to 1 recipients [VM 0x8001: 0x6 (offset 48)]): fully sent with 1 fragments, 1 retrieved, sender's original mode: 0x7 SHARE 0x2 (from VM 0x0, attributes 0x2f, flags 0x8, tag 0, to 1 recipients [VM 0x8001: 0x6 (offset 48)]): fully sent with 1 fragments, 1 retrieved, sender's original mode: 0x7 WARNING: Stage-2 page fault: pc=0x40032106, vmid=0x8001, vcpu=13, vaddr=0x40046ce0, ipaddr=0x8a9b28ce0, mode=0x81 0x63 NOTICE: Injecting Data Abort exception into VM 0x8001. D/TC:013 0 abort_handler:550 [abort] abort in User mode (TA will panic) E/TC:??? 0 E/TC:??? 0 User mode data-abort at address 0x40046ce0 (translation fault) E/TC:??? 0 esr 0x94020007 ttbr0 0x20000f03061a0 ttbr1 0x00000000 cidr 0x0 E/TC:??? 0 cpu #13 cpsr 0x00000130 E/TC:??? 0 x0 000000004003e348 x1 000000004003e349 E/TC:??? 0 x2 0000000040046ce0 x3 000000004003e348 E/TC:??? 0 x4 0000000040036088 x5 0000000000000000 E/TC:??? 0 x6 0000000000000000 x7 0000000040042e28 E/TC:??? 0 x8 0000000000000000 x9 0000000000000000 E/TC:??? 0 x10 0000000000000000 x11 0000000000000000 E/TC:??? 0 x12 0000000000000000 x13 0000000040042e28 E/TC:??? 0 x14 00000000400213cf x15 0000000000000000 E/TC:??? 0 x16 0000000000000000 x17 0000000000000000 E/TC:??? 0 x18 0000000000000000 x19 0000000000000000 E/TC:??? 0 x20 0000000000000000 x21 0000000000000000 E/TC:??? 0 x22 0000000000000000 x23 0000000000000000 E/TC:??? 0 x24 0000000000000000 x25 0000000000000000 E/TC:??? 0 x26 0000000000000000 x27 0000000000000000 E/TC:??? 0 x28 0000000000000000 x29 0000000000000000 E/TC:??? 0 x30 0000000000000000 elr 0000000040032106 E/TC:??? 0 sp_el0 0000000040042f80 E/LD: Status of TA f4e750bb-1437-4fbf-8785-8d3580c34994 E/LD: arch: arm E/LD: region 0: va 0x40006000 pa 0xf0404000 size 0x002000 flags rw-s (ldelf) E/LD: region 1: va 0x40008000 pa 0xf0406000 size 0x011000 flags r-xs (ldelf) E/LD: region 2: va 0x40019000 pa 0xf0417000 size 0x001000 flags rw-s (ldelf) E/LD: region 3: va 0x4001a000 pa 0xf0418000 size 0x004000 flags rw-s (ldelf) E/LD: region 4: va 0x4001e000 pa 0xf041c000 size 0x001000 flags r--s E/LD: region 5: va 0x4001f000 pa 0x00001000 size 0x017000 flags r-xs [0] E/LD: region 6: va 0x40036000 pa 0x00018000 size 0x00c000 flags rw-s [0] E/LD: region 7: va 0x40042000 pa 0xf0440000 size 0x001000 flags rw-s (stack) E/LD: region 8: va 0x40043000 pa 0x8b9101620 size 0x003000 flags rw-- (param) E/LD: region 9: va 0x40046000 pa 0x8a9b28ce0 size 0x001000 flags rw-- (param) E/LD: [0] f4e750bb-1437-4fbf-8785-8d3580c34994 @ 0x4001f000 E/LD: Call stack
Regards,
Yuye
------------------------------------------------------------------
发件人：梅建强(禹夜) <meijianqiang.mjq(a)alibaba-inc.com>
发送时间：2023年2月4日(星期六) 11:40
收件人：op-tee <op-tee(a)lists.trustedfirmware.org>
抄 送：赵哲(为哲) <weizhe.zz(a)alibaba-inc.com>; 高海源(码源) <haiyuan.ghy(a)alibaba-inc.com>; 王一蒙(北见) <wym389994(a)alibaba-inc.com>
主 题：OPTEE TA Crash
Hi,
Does anyone have a good solution to this problem？
https://github.com/OP-TEE/optee_os/issues/5803 <https://github.com/OP-TEE/optee_os/issues/5803 > <https://github.com/OP-TEE/optee_os/issues/5803 <https://github.com/OP-TEE/optee_os/issues/5803 > >
regards,
yuye
--
Hafnium mailing list -- hafnium(a)lists.trustedfirmware.org
To unsubscribe send an email to hafnium-leave(a)lists.trustedfirmware.org
