Writing $$external extensions with QLib

Anything QL Software or Programming Related.
HAOUI
Bent Pin Expansion Port
Posts: 89
Joined: Tue Dec 14, 2010 2:17 pm

Re: Writing $$external extensions with QLib

Post by HAOUI »

Hi all.

Let's go back to $$external programming for a last common side; the parameters passing.

Despite the fact that most interesting feature with externals is the share of BASIC compiled libraries, the most attractive use is the ability to turn easily a compiled BASIC PROCedure/FUNCtion to System Keyword, resident and re-rentrant code.

This naturally implies passing back and forth some parameters between the caller and the external. QLIB does support this parameters passing, at least with v3.36 and the right RUNTIMEs, but in a very limited, undocumented enough and obscur (badly) way - sorry.

As stated in the manual, "When called from interpreter ... only scalar parameters can be passed...". This is not really a big lack but if you try to pass an array as parameter, you will be notified by a "Not Implemented" error message while your system is probably already crashed. Remember, any QLIB run time error occuring within an external job drives directly to crash.

Also stated in the manual,"If the value of a parameter is altered by external ...then the corresponding variable in the caller program is also altered...". In fact, all passed parameters to external are copied back when job ends, altered or not. If an UNSET variable is passed to an external, on return, variable is SET anyway but this is not really dramatic either.

More impotant, Passing parameters to/from external is entirely carried out by QLIB RUNTIMEs, Embeded or Not. Thus, the RUNTIME used (version, mods, patches, ...) can have a certain influence on the way this is done when they are invoked on a target system. Better to have integrated a working RUNTIME in the external binary file, but it costs about 11 Ko whereas the external tokenized program can have a size of 2-3 Ko (large external extensions are not recommended).

I have no idea how RUNTIMEs handle parameters passing and I haven't managed to obtain any real help or valid information about this. I didn't want to hack RUNTIMEs either but I learned some useful rules which allowed me to realize some improvements in my work on this area.

Main things I have learned :
  • - paramaters are passed by value and not by reference
    - paramaters are not "casted" and still typed like originals from the caller, regardless of arguments type in the external job (pretty QDOS compliance)
    - Missed parameters are not created or passed neither by the caller nor by the RUNTIMEs or external job (missed parameters are created in S*BASIC)
    - Strangely enough, no entries are created in the Name Table of the external job for the passed parameters or their copies despite the fact External job uses its own independent table.
Direct consequences of this :
  • - Any string parameter has to be passed as quoted expression or a set string variable ended by $. This is boring for name parameters passing (e.g. filename)
    - All usual assembler staff used to manipulate parameters can't work (finding name of parameter per example) as no entries exist in the Name Table for these parameters.
    - The actual number of passed parameters (which becomes here variable) can't be known at external job starting.
    - All parameters flags are lost for good (e.g. # or separators , ; \ ! TO)
    - Furthermore, the TK2 PARNAM$() and PARSTR$() FUNCtions are not working here and they crash system if used. This is because TK2 lies on entries in the Return Table of S*BASIC which are not created by QDOS when calling machine-code or assimilated $$external code.
So, the most sensitive point when executing an external job is the use in any way whatsoever a supposed passed parameter which may was not really passed, or it was not a SET variable or it couldn't be coerced to the required type. This fire up immediately a QLIB RUNTIME error and the usual "Red Popup of Death".

The main precaution to avoid this is always "test existence and type of any parameter" before using in any affectation or expression in the external. TK2 PARUSE() and PARTYP() are very helpfully for this and still working within compiled externals. Also keep in mind that UNSET variables are now handled differently on QDOS and SMSQ/E systems and that QLIB compiled programs are more QDOS method compliant (BANG).

If you have reached this point of this long article (which may be not in the Forum spirit), you may find useful the attached archive containing my now released special toolkit PARAMEX I have written in assembler to deal with parameters passing for externals. It allows to fectch easily name or string parameters, to know how many actual parameters was passed and to find any hash or separator flags used along parameters. This is a "MINIMUM TO HAVE" for external programming.

The method I used is to access the Name Table of the caller, where all passed parameters are awaiting until the (happy) end of the executed external job. It was a very tenious but exciting task.

Also included some readme text and a BASIC external program you can use it as example to test what happens on your system about parameters passing.

As promised. Alain
PARAMEX.ZIP
(14.65 KiB) Downloaded 92 times


EmmBee
Trump Card
Posts: 240
Joined: Fri Jan 13, 2012 5:29 pm
Location: Kent

Re: Writing $$external extensions with QLib

Post by EmmBee »

pjw wrote:First things first: You are cleaning up the code at the moment. I dont want to disturb or duplicate that effort. When you feel ready, Id be happy to take a look and add any suggestions I might have, as will hopefully other interested parties. Once we are clear about whats going on and how it works, then would be a good time to start improving things, IMHO.
I've now completed my efforts, renaming Martin's Q_Liberator decompile. This was posted on the decompiler thread. So, I thought I would also post it here in case you missed it and for everyone else who would like to see this. I believe I have got the vast majority of the meanings correct. There were just under 400 variable names to attach the right meaning to. I welcome any comments.

Now that we have the basic source code, it should be quite easy for us to maintain, that is to cure any bugs that people come across, and also to make extensions. One feature I would like to see would be the ability to add a REMark to the top line of any multi-structure. This can be done in both QDOS and SMSQ/E, but currently not for QLIB and Turbo, when the top line changes to become a single line construct. If anybody would like to work out how this can be done, then please go ahead!
Here is the code ..
QLIB_BAS_english.zip
(26.86 KiB) Downloaded 92 times
Also, I thought Marcel wrote that he already had disassembled much of the machine code.. No point in duplicating the effort, especially since the best possible effort may already have been done..
Indeed, Marcel told us he had been using a state-of-the-art disassembler. I look forward to seeing his work, eventually.


User avatar
RalfR
Aurora
Posts: 872
Joined: Fri Jun 15, 2018 8:58 pm

Re: Writing $$external extensions with QLib

Post by RalfR »

Does the way to start externals via "!" now works (if the several patch programs from WLs site are used)?


4E75 7000
EmmBee
Trump Card
Posts: 240
Joined: Fri Jan 13, 2012 5:29 pm
Location: Kent

Re: Writing $$external extensions with QLib

Post by EmmBee »

tofro wrote:
HAOUI wrote:
tofro wrote:
That is really a nice idea to build a self-contained multi-threaded S*BASIC program, however:

It should, but it doesn't, - At least not on SMSQ/E.
I used to have an idea why not, but forgot...

Tobias
Indeed, this often crashed my systems. My be related to copying back parameters or multiple versions, mods and patchs.
A simple rule I learned : Have a normal, good & successful return from QLIB program or GOTO reset button ;o)
Alain
If I remember right, that wasn't the problem - I think an ! command simply didn't spawn a new job on SMSQ/E.

Tobias
I have tested this by writing a simple external program that just prints out a message and then goes into an infinite loop.

Using QPC2, and starting such an external from the Interpreter with PROG! does seem to work for me, and as stated in the manual. However, I find that Ctrl-Space and Ctrl-c need to be pressed to get the Interpreter up and running again, while the external job continues to run.

Instead, when I start this same external without the "!", then the job does run as expected, and the Interpreter remains suspended. Pressing Ctrl-Space will terminate that job. All of this is expected.

I am using what I believe to be all the latest patches.
Last edited by EmmBee on Fri Feb 21, 2020 11:54 pm, edited 1 time in total.


User avatar
RalfR
Aurora
Posts: 872
Joined: Fri Jun 15, 2018 8:58 pm

Re: Writing $$external extensions with QLib

Post by RalfR »

EmmBee wrote:I am using what I believe to be all the latest patches.
Which ones? There are so many....


4E75 7000
EmmBee
Trump Card
Posts: 240
Joined: Fri Jan 13, 2012 5:29 pm
Location: Kent

Re: Writing $$external extensions with QLib

Post by EmmBee »

Ralf R. wrote:
EmmBee wrote:I am using what I believe to be all the latest patches.
Which ones? There are so many....
I usually visit Dilwyn's website for most of my QLIBs. However, once there, I would not advise searching for 'QLIB' because you'll be taken to the wrong page! Instead, you should scroll down and look on the right-hand side. You will eventually see the QLIB logo and a link to click on. That should take you to the QLiberator section. About half-way down that page, you'll find "QLIB_RUN modified 2" - that's the one I recommend. The ERLIN and ERNUM functions are the correct way round, also the $externals will work in SMSQ/E.


EmmBee
Trump Card
Posts: 240
Joined: Fri Jan 13, 2012 5:29 pm
Location: Kent

Re: Writing $$external extensions with QLib

Post by EmmBee »

HAOUI wrote:NEXT_TOKEN is a more complex code as it contains a table of offsets (14 entries) and jmp is made to one of these locations.
So we have to pull out all these jumped codes also. This new archive contains the complete assembler for NEXT_TOKEN.
Alain
DeaQlib2.zip
Hi Alain
Thanks for this code. We are needing to add a further 4 cases. These are for hex%, hex0%, bin%, bin0%. The ascii codes for these are: 216, 208, 232, 224
These new codes need to be treated exactly the same as for floating-point numbers. The actual values needed are exactly the same as for floating-point.
If I were writing this in SuperBASIC, I could write .. value = PEEK_F(src) : src = src+6, and then return the number - for all of them!
Is it possible for you to make these changes, please?
Kind regards,
Michael


User avatar
RalfR
Aurora
Posts: 872
Joined: Fri Jun 15, 2018 8:58 pm

Re: Writing $$external extensions with QLib

Post by RalfR »

EmmBee wrote:
Ralf R. wrote:
EmmBee wrote:I am using what I believe to be all the latest patches.
Which ones? There are so many....
I usually visit Dilwyn's website for most of my QLIBs.
I meant the patch programs from WLs site. There are a few with similar names.


4E75 7000
EmmBee
Trump Card
Posts: 240
Joined: Fri Jan 13, 2012 5:29 pm
Location: Kent

Re: Writing $$external extensions with QLib

Post by EmmBee »

Ralf R. wrote:
EmmBee wrote:
Ralf R. wrote: Which ones? There are so many....
I usually visit Dilwyn's website for most of my QLIBs.
I meant the patch programs from WLs site. There are a few with similar names.
Can you give us a link?


User avatar
RalfR
Aurora
Posts: 872
Joined: Fri Jun 15, 2018 8:58 pm

Re: Writing $$external extensions with QLib

Post by RalfR »

EmmBee wrote:
Ralf R. wrote:
EmmBee wrote: I usually visit Dilwyn's website for most of my QLIBs.
I meant the patch programs from WLs site. There are a few with similar names.
Can you give us a link?
Direct link for the ZIP is: https://www.wlenerz.com/smsqe/utilities.zip

I have thought, these ones are for repairing the things.


4E75 7000
Post Reply