Introducing File Items

File Item
INTRODUCING	NEW	SET	ATTRIBUTES	EXAMPLES

The STx FILE item can be used to access plain text files, XML files, and STx proprietary binary files. It can also be used to to query the status of files and directories on disk.

Text Files

STx text file items can be used to read and write ASCII and UNICODE text files.

Text file items are created with the option /Text.

Reading and Writing to Text Files

The commands READ and WRITE, and the file item commands LOAD and SAVE can be used to read from and write to text files.

Text File Example

The following script demonstrates the use of a text file item.

[macro text_file_example]
    //
    // create file item
    //
    #fname := set 'text_file_example.txt'
    #f := new file * '$#fname' /Text /Write
    if '$#f[?]' != 'file' then
        em -1 ; '$#mac - Create file failed ($EMSG)'
    end
    //
    // write data to file using 'write' command
    //
    write $#f 'The first line of this text file' /Newline
    if '$rc' > 0 em $rc ; '$#mac - Failed to write line to file'
    write $#f '%d:%d:%d' 1 2 30 /Format /Newline
    if '$rc' > 0 em $rc ; '$#mac - WRITE /Format failed ($EMSG)'
    //
    // write data to file using 'save' command
    //
    $#f save $(eval fill(10,0,1))
    //
    // close file and reopen as read-only
    //
    delete $#f
    #f := new file * '$#fname' /Text /Read
    if '$#f[?]' != 'file' then
        em -1 ; '$#mac::Error - Create file failed ($EMSG)'
    end
    //
    // load data from file into a table and display the table
    //
    #t := new table *
    if '$#t[?]' != 'table' then
        em -1 ; '$#mac - Create table failed ($EMSG)'
    end
    $#f load $#t
    if '$rc' > 0 em $rc ; $#mac - LOAD failed ($EMSG)
    showitem $#t
    //
    // clean up
    //
    delete $#f $#t
exit

Section Files

Section files are text files (ASCII or UNICODE) which are split up into different sections using the following format:

[sectiontype sectionname] data [sectiontype sectionname] data

This format was used to store all STx settings prior to version 3.0.

A section file is created with the option /Section.

Section File Example

The following script demonstrates the use of a section file item.

[macro section_file_example]
    //
    // create the section file item
    //
    #fname := set 'section_file_example.txt'
    #f := new file * '$#fname' /Section /Write
    if '$#f[?]' != 'file' then
        em -1 ; '$#mac - Failed to create the section file item'
    end
    //
    // create test data in a simple table
    // and write to the section file
    //
    #t := new table *
    $#t * '0 1 2 3 4 5 6 7 8 9'
    $#t * '10 11 12 13 14 15 16 17 18 19'
    // The section will be called "section1", and its type will be "section". Hence, the
    // section header that gets written to the file will be "[section section1]".
    $#f save $#t section section1
    //
    // save the same data to another section
    //
    // The section will be called "section2", and its type will be "section", too. Hence, the
    // section header that gets written to the file will be "[section section2]".
    $#f save $#t section section2
    //
    // save the same data to yet another section, this time a section of different type
    //
    // The section will be called "section3", and its type will be "Cappuccino". Hence, the
    // section header that gets written to the file will be "[Cappuccino section3]". (This is
    // to demonstrate that the section type may be an arbitrary string.)
    $#f save $#t Cappuccino section3
    //
    // The section will be called "section3", too, and its type will be "section". Hence, the
    // section header that gets written to the file will be "[section section3]". This demonstrates
    // that section names need not be unique across section types.
    $#f save $#t section section3
    //
    // open file using default text editor
    //
    system open '$#fname'
    //
    // list all sections in the file
    //
    if '$($#t /Delete)' > 0 then
        em $rc ; $#mac - Failed to empty table contents ($EMSG)
    end
    $#f list $#t
    if '$rc' > 0 em $rc ; $#mac - LIST failed ($EMSG)
    showitem $#t
    //
    // extract data from one of the file's sections
    //
    if '$($#t /Delete)' > 0 then
        em $rc ; $#mac - Failed to empty table contents ($EMSG)
    end
    $#f load $#t section section1
    if '$rc' > 0 em $rc ; $#mac - LOAD failed ($EMSG)
    readtab $#t 0 #0 #1 #2 #3 #4 #5 #6 #7 #8 #9
    if '$rc' > 0 em $rc ; $#mac - Failed to read table data
    butil 'msgbox msg ; The variable #1 contains the value "$#1"'
    //
    // delete file item
    //
    delete $#f
exit

Status File

An STx status file item can be used to query the status of a file on disk, e.g. if it exists, when it was last modified etc. It cannot access the content of the file.

A status file item is created using the option /File.

Status File Example

The following script demonstrates the use of a status file item.

[macro status_file_example]
    readvar #argv #filePath
    //
    // create the status file item
    //
    #f := new file * /File
    if '$#f[?]' != 'file' then
        em -1 ; '$#mac - Failed to create the status file item'
    end
    //
    // interpret #argv parameter as a file path or ask user to choose 
    // a demonstration file
    //
    if '$#filePath' == '' then
        butil 'FileDialog open ; Select any file'
        #filePath := set $result
        if '$#filePath' == '' exit      
    end
    //
    // check file exists
    //
    $#f status $#filePath
    if '$rc' != 0 em $rc ; 'msgbox msg ; "$#filePath" does not exist'
    butil 'msgbox msg ; The file "$#filePath" exists'
    //
    // display last modification date/time
    //
    readstr '$#f[!modified]' #date #time
    butil 'msgbox msg ; Last modified on $#date at $#time'
    //
    // ask user if they want to delete the file
    //
    butil 'msgbox yesno ; Delete "$#filePath" from disk?'
    if '$result' == 'yes' $#f delete $#filePath
    //
    // delete the status file item (not the file)
    //
    delete $#f
exit

XML Files

STx file items can also read and write XML files. The following is the content of a simple XML file:

XML File Example

The following script demonstrates the use of an XML file item.

[macro xml_file_example]
    readvar #argv #example';'#argv /Delete
    if '$#example+0' != '$#example' #example := 1
    if '$#example' < 1 || '$#example' > 2 #example := 1
    // create the XML file
    #f := new file * /XML /Write
    if '$#f[?]' != 'file' then
    	em -1 ; '$#mac::Error - Failed to create the XML file item'
    end
    // do the example
    GoSubX Example_$#example $#argv
    // delete file item
    delete $#f
exit
//
// Create some elements and attributes, modify them and display them
// Uses the commands:
//      root | addelement | goto | find | deleteelement | save
//
Example_1:
    // create XML data
    //
    //   create root
    //
    $#f root 'rootTag' attr1 attrValue1 /Delete
    if '$rc' > 0 em $rc ; $#mac - Failed to set root tag
    //
    //   create elements
    //
    $#f goto /In
    if '$rc' > 0 em $rc ; $#mac - Failed to go into root element
    for #i := 0 to $#i < 10 step #i := int $#i + 1
        $#f addelement 'elemTag' ID $#i
        if '$rc' > 0 then
            em $rc ; $#mac - Failed to add XML element "elemTag ID $#i"
        end
        $#f setdata '$(eval fill(10,0,$#i))' /Table /Numeric /Vector
        if '$rc' > 0 then
            em '$rc ; $#mac - Failed to set the element`'s data'
        end
    end
    //
    // find and delete the element with the ID 3
    //
    $#f find * 'ID:==:3' /First
    if '$rc' == '0' $#f deleteelement
    //
    // save XML file
    //
    $#f save 'xml_file_example_1.xml'
    if '$rc' > 0 then
    	em $rc ; $#mac - Failed to save "xml_file_example_1.xml"
    end
    //
    // display the file
    //
    showitem $#f
exit
//
// Ask user to select an STx project file, check if it is a valid project file and
// find the longest file. Uses the XML file item commands:
//      load | goto | extracttable | position
//
Example_2:
    readvar #argv #fDatasetPath
	#pwd := pwd // save working directory
    if '$#fDatasetPath' == '' then
        // ask user to select dataset file from disk
        #caption := set 'Select an STx project from disk'
        #filetype := set 'STx Project File'
        BUtil FileDialog Open ; $#caption ; $@WORK ; stxpr=$#fileType
        readvar result #fDatasetPath
		pwd $#pwd // restore working directory
        if '$#fDatasetPath' == '' exit
    end
    //
    // check if it is a valid dataset file
    //
    $#f load '$#fDataSetPath'
    if '$rc' > 0 then
    	em $rc ; $#mac - Failed to load the file "$#fDatasetPath".
    end
    $#f goto '/STXDataSet'
    if '$rc' > 0 em $rc ; $#mac - Invalid STx project file
    if '$#f[!xattr,AFile]' == 'Link' then
    	#msg := set 'The project you chose is linked and contains no'
    	#msg := set $#msg ' segments. Please choose an unlinked project.'
        butil 'msgbox msg ; $#msg'
        exit 1 set '$(xml_file_example 2)'
    end
    //
    // extract all segments from the dataset into table
    //
    #tExt := new table * * /E pos tag id length
    if '$#tExt[?]' != 'table' then
        em -1 ; $#mac - Failed to create extended table
    end
    $#f extracttable $#tExt pos ASeg tag ID id L length /Recursive
    if '$rc' > 0 then
    	em $rc ; $#mac - EXTRACTTABLE failed ($EMSG)
    end
    showitem $#tExt ; All segments in the project "$#fDataSetPath"
    //
    // find and select the longest segment
    //
    #max := 0
    for #i := 0 to $#i < $#tExt[] step #i := int $#i + 1
        #len := $#tExt[$#i,length]
        if '$#len' > '$#max' then
            #max := $#len
            #iMax := $#i
        end
    end
    $#f position $#tExt[$#iMax,pos] /Goto
    if '$rc' > 0 then
        em $rc ; $#mac - Failed to select the longest segment
    end
    // show file
    showitem $#f
    // clean up
    delete $#tExt
exit

Binary Files

Binary file items are supported from STx version 3.7.x (r603). The following features are available:

• A binary file can have one or more sections.
• Every section can have a numeric item saved in it (e.g. a table, a value etc.).
• A section has a header with the following attributes:
  • The data type (16-bit integer, 32-bit integer, 32-bit float or 64-bit float).
  • The size of the value in bytes (2, 4 or 8).
  • The number of lines.
  • The number of columns.
• The file item implements functions to load and save data to sections and to list existing sections.
• Existing sections can be overwritten by data of the same length. If data is not the same length, the section must be deleted first.
• Individual sections or all sections can be deleted (see NEW file and SAVE).
• Sections can be appended to an existing file.

The STx binary file contains no metadata describing the section data! Metadata, if needed, must be stored elsewhere (e.g. in the project file).

Binary File Format

The STx binary file has the following format:

Binary Header

The binary file section header is 12 bytes long and contains three 32 bit integers.

header[0]

the binary file type:

0x0102 - 16bit integer

0x0204 - 32bit integer

0x0304 - 32bit float

0x0408 - 64bit float

header[1]

the number of rows

This is stored in a signed integer.

header[2]

the number of columns

This is stored in a signed integer.

Binary Data

The binary data is stored directly after the binary header. Note that each column is stored as a vector. The column vectors are stored one after the other.

Binary File Example

The following script demonstrates the use of a binary file item.

[macro binary_file_example]
    //
    // create binary file item
    // deleting existing contents
    //
    #f := new file * 'binary_file_example.stb' /Binary /Delete
    if '$#f[?]' != 'file' em -1 ; $#mac - Failed to create #f
    //
    // save vector data to section 0 (next free section)
    //
    $#f save $(eval fill(10,1,1)) 
    //
    // save matrix data to section 1
    //
    #matrix := eval init(10,10,0)
    showitem $#matrix
    $#f save $#matrix
    if '$rc' > 0 em $rc ; SAVE failed ($EMSG)
    //
    // save matrix data to section 2
    //
    $#f save $(eval init(10,10,1))
    //
    // list sections in binary file
    //
    #st := new table *
    if '$#st[?]' != 'table' em -1 ; $#mac - Failed to create #st
    if '$($#f list $#st)' > 0 em $rc ; $#mac - LIST failed ($EMSG)
    showitem $#st
    //
    // delete section 1
    //
    $#f save * 1
    //
    // load matrix data from section 1 (was section 2)
    //
    #t := new table * 10 /P
    if '$#t[?]' != 'table' em -1 ; $#mac - Failed to create #t
    if '$($#f load $#t 1)' > 0 em $rc ; $#mac - LOAD failed ($EMSG)
    showitem $#t
    //
    // clean up
    //
    delete $#f $#matrix $#st $#t
exit

Here's a second example file (binaryfile.sts) which can be found in the script examples folder.

[Macro BinaryFile]

//	set testfile path and delete existing testfile
#path := '$scriptdirectory\binaryfile.dat'
filetoolbox delete '$#path'

con log '$#mac: test binary file write, file=$#path'
//	create and open the binary file
#file := new file * '$#path' /b
if $rc != 0 con elog '2;;open binary file "$#path" failed'
//	save a vector, use default format (/4 = double)
#data := eval fill(100,0,1)
$#file save $#data
con log 'write double vector with $#data[!nrow] elements: RC=$rc, section=$#file[!bininfo]'
//	save a matrix, use integer format (/2 = 32 bit integer)
#data := eval vmcol($#data, 100-$#data)
$#file save $#data /2
con log 'write short integer matrix with $#data[!nrow] rows and $#data[!ncol] columns: RC=$rc, section=$#file[!bininfo]'
delete $#file $#data
con log 'write test finished - file closed'

con log '$#mac: test binary file read, file=$#path'
//	open the binary file in read only mode and create i/o tables
#file := new file * '$#path' /b/r
if $rc != 0 con elog '2;;open binary file "$#path" failed'
#list := new table *
#data := new table *
//	read section list and all section data
$#file list $#list
con log 'load section list: RC=$rc, sections=$#file[!bininfo]'
for #i := 0 to $#i < $#list[] step #i := int $#i+1
	con log 'section header for section $#i = $#list[$#i]'
	$#file load $#data $#i
	con log 'result of load section $#i: rows=$#data[!nrow], columns=$#data[!ncol], RC=$rc, section=$#file[!bininfo]'
end
delete $#file $#list $#data
con log 'read test finished - file closed'
con log '$#mac: finished - press Continue or Exit to close console window'

//	done
exit

R Script - Read STx Binary File Example

This R script can read the contents of an STx binary file into R.

# Read vectors and matrices from {{Stx}} binary files.
# The output is a list of all vectors/matrices.
# They can be accessed by using [[index]].
# Note that this only works for matrices and
# vectors with double values.
#
# The following example shows how a binary file with
# two entries can be read:
#
# data <- read.binary.file("example.bin")
# firstentry <- data[[1]]
# secondentry <- data[[2]]

read.binary.file <- function(filename) {
 # open connection to binary file
 con <- file(filename, "rb")

 # counter
 index <- 0

 # this list will contain the names of the variables
 tablelist <- NULL

 # read the first header
 header <- readBin(con, integer(), 3)

 # while there is a header ...
 while (length(header) >0) {
   index <- index + 1
   nrows <- header[2]
   ncols <- header[3]
     # read the current matrix
   data <- matrix(readBin(con, double(), nrows*ncols), nrow=nrows, ncol=ncols)
     # the variable name of the current matrix
   tablename <- paste("index", index, sep="")
     # assign the current matrix to the variable
   assign(tablename, data)
     # store the variable name in a vector
   tablelist <- c(tablelist, tablename)
     # save some memory
   rm(data)
     # read the next header
   header <- readBin(con, integer(), 3)
 }

 # close connection to binary file
 close(con)

 # create a list of the read matrices
 outputlist <- list()
 for (i in 1:length(tablelist)) {
   outputlist <- c(outputlist, list(eval(parse(text=tablelist[i]))))
 }

 return(outputlist)
}

GDX Files

GDX stands for "graphics data exchange" and is a binary file format that a graph item can display in a plot.

The GDX format was introduced to STx in version 3.9.0