QL / PASCAL

[NormanDunbar]

I think I need Marcel's help here, I might have a new bug in SMSQ!

TL;DR:

There may be a bug in SMSQ where FS_POSAB/IOF_POSA is called to set a new position beyond EOF for a file. The new file position returned in D1 is always zero in this case, and subsequent calls to FS_POSRE/IOF_POSR with an offset of zero also return zero as the current file position in D1. This problem does not occur when FS_POSAB/IOF_POSA is called with a new position which is within the bounds of the file.

If the file is 100 bytes long, calling FS_POSAB/IOF_POSA with a position of anything between 0 and 100 will return D1 correctly set. Subsequent calls to FS_POSRE with an offset of zero return the current position.

If the call to FS_POSAB/IOF_POSA is greater than 100, D1 is returned as zero, the file's new position is not set at EOF, D0 is returned set to ERR_EF (-10) to indicate EOF, but the file position doesn't change. Subsequent calls to FS_POSRE/IOF_POSR return zero.

The above assumes that the file is open in mode 0, OLD_EXCLUSIVE mode.

Gory Details:

The Pascal Append() function opens the file in Q_OPEN mode, (=0 = OPEN in SuperBASIC), which is fine. It then calls fs_posab/iof_posa to set the file position to $FFFFFFFF which works, but as this is beyond EOF, it returns D1 set to 0 and not as the new file position. It then calls fs_posre/iof_posr with an offset of 0 to get the current file position.

Scenario 1:

When the fs_posab/iof_posa new position is within the bounds of the file, D1 returns with the new file position correctly set. If I request position 2, it comes back from fs_posab/iof_posa set to 2. When I subsequently call fs_posre/iof_posr with an offset of zero, it comes back with D1 = 2. All is well and correct.

Scenario 2:

When the fs_posab/iof_posa new position is past the current EOF, D1 returns with the value zero. This is not correct!. When I subsequently call fs_posre/iof_posr with an offset of zero, it comes back with D1 = 0. All is not at well and is incorrect.

So, this small Pascal file:

Code: Select all

{ Program to demonstrate the Append function. }

Var f : text;

begin
  Assign (f,'ram1_test.txt');

  Rewrite(f);
  Writeln(f, 'This is the first line of text.');
  close(f);

  Append(f);
  Writeln(f,'This is the second line of text.');
  close (f);
end.

If I let it run unmolested, it will open the file in mode 3 (Q_OPEN_OVER), write the first line and close the file. It then reopens the file in Q_OPEN mode (zero), requests a new position of $FFFFFFFF, then does a relative position of zero and gets a result of zero. The file is then written from the start, overwriting the first line of text.

If, on the other hand, I use QMON to break into setting the fs_posab position, and change D1 from $FFFFFFFF to $10, for example, or any other valid "in bounds" value, the text is correctly appended at that position.

I had a suspicion that Pascal was making a call to fs_truncate with the position returned from fs_posre but I tested this and it doesn't call fs_truncate, so that went out the window. I knocked up an assembly test instead:

Code: Select all

Open    moveq #io_open,d0
        moveq #-1,d1
        moveq #3,d3          ; OPEN_OVER
        lea file,a0
        trap #2
        nop

Reset   moveq #io_sstrg,d0
        moveq #11,d2
        moveq #-1,d3
        lea digits,a1
        trap #3
        nop

Close_1 moveq #io_close,d0
        trap #2
        nop

ReOpen  moveq #io_open,d0
        moveq #-1,d1
        moveq #0,d3          ; OPEN
        lea file,a0
        trap #2
        nop

Posab   moveq #fs_posab,d0
        moveq #-1,d1
        trap #3
        nop

Posre   moveq #fs_posre,d0
        moveq #0,d1
        trap #3
        nop

Append  moveq #io_sstrg,d0
        moveq #27,d2
        moveq #-1,d3
        lea letters,a1
        trap #3
        nop

Close_2 moveq #io_close,d0
        trap #2
        nop

Exit    rts

file    dc.w file_end-file-2
        dc.b 'ram1_test'
file_end equ *

digits  dc.b '1234567890',$0a

letters dc.b 'AbcdeFghijkLmnopQrstuVwxyz',$0a

A similar program, in SuperBASIC using SET_POSITION and FILE_POSITION from DJToolkit shows exactly the same problem. But they use FS_POSAB/IOF_POSA and FS_POSRE/IOF_POSR under the covers, so I'm not surprised.



When I traced this through, I got the same problem, fs_posab/iof_posa returned zero in D1, fs_posre/iof_posr did the same, and the only text in the file were the letters. I think we have a bug in SMSQ (or QPC's version) where D1 is not being correctly returned when a file is opened, positioned at EOF using fs_posab/iof_posa with D1 set to $FFFFFFFF, then calling fs_posre/iof_posr with a relative position of zero to get current file position. I suspect fs_posab/iof_posa is not setting the new file position if EOF errors are reported?


Cheers,
Norm.

[tofro]

Norman,

Notwithstanding that I really think the file position returned by FS.POSAB should be "where the next write is going to put a byte" in all cases, (and I doubt that is the case here - Can you check where a byte output with IO.SBYTE is actually ending up after this return?)

I'm not so sure whether the behavior you observe is what people would expect from FreePascal's "Seek" procedure. If I remember right, a Seek beyond EOF on a writable file should actually extend the file with zero bytes until that position is valid (i.e. emulating a Unix sparse file).

[Andrew]

If I remember right, a Seek beyond EOF on a writable file should actually extend the file with zero bytes until that position is valid (i.e. emulating a Unix sparse file).

If I remember correctly in Turbo Pascal, Borland Pascal and Delphi a seek beyond EOF will result in an error. To expand the file, you can move the pointer one record past the last record in the file using Seek (F, FileSize (F)) or by performing a SeekEOF.

[NormanDunbar]

If we ignore the Pascal thing for now, I put up some callable assembly code which should open a file, write to it, close it, open again, position at EOF and write more text and finally close the file. I also tested a Basic program to do the same.

Both fail.

As long as the file position is beyond EOF, it doesn't move the current file position.

I think it should, and I'm sure it used to. I'm not near Pennel or the docs ATM, but when I tested DJToolkit's ABS_POSITION (I think!), it worked. My code cleared the ERR_EF which occurred, but I'm sure the file pointer was set to EOF ready to append.

Now there's no appending. The "appended" text is written at the start. (Or wherever the current file position happens to be.)

Cheers,
Norm.

[janbredenbeek]

If we ignore the Pascal thing for now, I put up some callable assembly code which should open a file, write to it, close it, open again, position at EOF and write more text and finally close the file. I also tested a Basic program to do the same.

Both fail.

As long as the file position is beyond EOF, it doesn't move the current file position.

I think it should, and I'm sure it used to. I'm not near Pennel or the docs ATM, but when I tested DJToolkit's ABS_POSITION (I think!), it worked. My code cleared the ERR_EF which occurred, but I'm sure the file pointer was set to EOF ready to append.

Now there's no appending. The "appended" text is written at the start. (Or wherever the current file position happens to be.)

You should avoid setting the file position to very high values. $FFFFFFFF might be interpreted as -1 and thus leave the position at the beginning.

I used to use 999999 to set the position to EOF, though this may be a bit on the low side nowadays.

I did a quick test using the ram driver. It correctly sets the position to the end when I specify 1024*1024*1024 (1GB) as position. Using values beyond 2^31-65 failed to set the position correctly and overwrote the first bytes of the file (the -65 is not really a surprise as this is one less than the size of the file header which is prepended to the stored file's contents).

Note that every directory device driver may interpret this its own way. The MDV driver splits the position in a 512-byte block number and byte position within block, each of which use 16 bits. Thus setting a position beyond 4MB will cause the block number to overflow when using signed arithmetic. The FLP driver probably uses the same logic (well, files couldn't grow bigger than 3.2MB anyway ).

All in all, it would have been better if the FS.POSAB/POSRE calls would have allowed setting the position from the end as well as the start of the file (perhaps by specifying negative numbers as parameter)...

Jan

[pjw]

I havent followed this discussion in detail (due to circumstances), but I had to react to some assertions made:
The max file size on a WIN device under SMSQ/E is 2^31b (2Gb), including the 64b header.
So the max legal position one could set is 2^31-65. Eg

Code: Select all

ch = fopen(<filename>): get#ch\2^31-65: print fpos(#ch): close

will print the file length of any file, demonstrating that the file pointer is set to the last possible position in a file.
(Of course, you couldnt write anything more to a file if the actual size was 2^31-65!)
If you increase the value in the example above to, ie, get#ch\2^31-64, the file position is returned as 0, ie rubbish.

[NormanDunbar]

Thanks Jan, Per. This is interesting.

I wasn't aware that file positions could be signed, for absolute positioning. It makes sense to use signed offsets for relative positions though.

The Pascal code must be assuming the same as it passed -1 to go to the end of the file. I'll have a play later.

Cheers,
Norm.

[NormanDunbar]

Interestingly, Pennel states that for relative positioning D1 is signed, but says nothing at all about whether is is/should be signed for absolute positioning. I may have to try and find the source code and see what's what! Maybe!


Cheers,
Norm.

[NormanDunbar]

So the max legal position one could set is 2^31-65

I've tested this in SuperBASIC with DJToolkit, and using 2^31-65 as the offset, I do indeed get the file position set to the last byte of the file. Any higher, and the file position remains at zero (for a newly opened file, with existing content obviously!). Right, now to test what Pascal does.......

Thanks again.


Cheers,
Norm.

[NormanDunbar]

Sunday's Pascal distractions, with thanks to Per and Jan for advice which solved a couple of problems.

• Fs_posab, fs_posre, fs_truncate and fs_headr were accessing parameters in the wrong order and overwriting some.
• Reset(), Rewrite() functions now have correct file open modes.
• Append() also, but this function was always writing to the start of the file, now fixed.

So, Append() was hitting runtime errors whenever it was used. I tried tracing this and discovered that when it called out to fs_posab to set the position to EOF, it was hitting an invalid channel id error. This was caused by the register parameter passing which passed the first two ordinal (numbers) parameters in D0 and D1. D0 was being overwritten by the FS_POSAB constant before it was passed to A0 as the channel id. Fixing this still hit run time errors, but different ones.

Checking all the other assembly functions for QL file handling etc showed that FS_POSRE, FS_TRNCATE and FS_HEADR needed fixing as well. Doing this stopped Append() from barfing at runtime. Hooray!

Append() was, however, writing the appended text to the start of the file, not the end. Thanks to Jan and Per, it appears that while D1 can be set to a very large number, it might be being interpreted as a signed number. Edit both Jan and Per advised that $7FFFFFBF is the largest file offset that can be used in a call to FS_POSAB. Testing showed this to be the case and anything higher will leave the file position where it is, and return ERR_EF.

The do_seekend() function in "sysfile.inc" was amended to use this offset when calling do_seek(), instead of using -1 as it was previously. Do_seek() calls down to fs_posab() in "qdos.inc" and this fixed the problem of not moving the file pointer to the end of file.

FS_POSAB in the "qdos.inc" file was also corrected to remove ERR_EF if it occurred. It turned out that after that fix, Append() still caused a runtime error 100 -- End of file. This was traced to FS_POSRE which is used by do_seekend() to get the current file position -- although, with the above fix, this shouldn't be required now.. It seems that if the current file position is already at EOF then FS_POSRE, even with an offset of zero, will still return ERR_EF. That too has been cleared and Append() works correctly now.

The attached file is a patch that can be applied to the source code in "<installation>/rtl/sinclairql" to bring everything up to where I am today.

FPC.patch.3.zip

(3.54 KiB) Downloaded 48 times

Cheers,
Norm.