Editing RangeOfFields

Printing a range of fields - all fields but the first, for examples, or fields 3 through 8 - is a surprisingly fiddly little problem. ==== No field offsets are stored Although awk performs field splitting, it does not maintain a record (or at least not that is accessible to user code) of the offsets into the line where the splitting actually took place. This means that you can, for example, assign an empty string to the fields preceding and following the range that you want ({{{ {$1 = $2 = ""; for (i = 9; i <= NF; i++) $i = ""; print} }}}). This will however cause awk to recompute the line adding OFS between the field, including the now empty fields at the beginning and at the end, in the case of the above example with a default FS and OFS, the spaces will be squeezed and 2 space will be present in front of all the line. You can remove the front space with {{{ $0=substr($0, 1+length(OFS) * 2) }}}, another possibility is to shift the fields and adjust NF eg to keep the fields 3 to 8: {{{ awk for (i=3;i<=8;i+=1)) $(i-2)=$i # shift $1=$3 $2=$4 ... NF=8-3+1 # only keep the first six fields print }}} ==== Drawbacks of using a loop In some cases, the desired behaviour is to remove everything before a certain field and then print out the rest of the line. The typical tactic is then to use a for loop, such as this: {{{ bash awk '{sep="";for (i=2;i<=NF;i++) {printf "%s%s",sep, $i;sep=" "}; printf "\n"}' file # or, avoids using sep awk '{for (i=2;i<=NF;i++) {printf "%s%s",(i>2?" ":""), $i}; printf "\n"}' file # uses OFS and ORS awk '{for (i=2; i<=NF; i++) printf("%s%s", $i, i==NF ? ORS : OFS)}' file }}} A loop to select individual fields will also cause anything that appears between the fields to be replaced with " ". This is often not the desired behaviour. ==== Using sub() If the separator is the default, you can use a direct sub() on $0 to remove the fields that you don't want. This has the advantage that original spacing between fields will be preserved. For example: {{{ bash awk '{sub(/^[[:blank:]]*([^[:blank:]]+[[:blank:]]+){n}/,"")}' file }}} will remove the first "n" fields from the line ("n" must be replaced by the actual number of fields that you want to remove). If you want to remove the last n fields with the same technique, then {{{ bash awk '{sub(/([[:blank:]]+[^[:blank:]]+){n}[[:blank:]]*$/,"")}' file }}} should do the job. If you want to keep from field n to field m (meaning "remove from field 1 to n-1 and from m+1 to NF"), just combine the above two techniques, using the appropriate values for the repetition operator. Keep in mind that the {n} repetition operator in regexes, while specified by POSIX, is not supported by all implementations of awk. With GNU awk, you have to use the --re-interval command line switch (or --posix to get full POSIX compatibility, which includes repetition operators). If FS is not the default, but it's still a single character (for example, "#"), it is simpler and you can do something like {{{ bash awk '{sub(/^([^#]*#){n}/,"")}' file }}} the example that removes the last n fields can be adapted similarly from the code that uses the default FS. Finally, if FS is a full regular expression, then the problem is not trivial and it's better to use some other technique among those described here. ==== Using cut If the field separator is a single character, the {{{cut}}} utility may be used to select a range of fields. There is documentation for the [[http://www.gnu.org/software/coreutils/manual/html_node/cut-invocation.html|GNU version of cut]], which appears in the coreutils package, but which leaves out mention of the range feature. The [[http://www.opengroup.org/onlinepubs/000095399/utilities/cut.html|Open Group spec]] describes cut in more detail, and the [[http://www.opengroup.org/onlinepubs/000095399/utilities/cut.html#tag_04_31_17|Examples]] section offers a more detailed guide to the syntax. {{{cut}}} cannot use field delimiters that are longer than a single character. By default, the delimiter is the tab character. To select fields one, three, and from field 9 to the end of the line, one could write: {{{cut -f1,3,9-}}} . Fields cannot be reordered in this way: {{{cut -f3,1,9-}}} is equivalent to the previous example. If a single-character delimiter limitation is not a restriction, and if the fields do not need to be reordered, and if no other computation needs to take place, consider using {{{cut}}} instead of {{{awk}}}: it is small, simple and fast, and this simplicity makes its purpose immediately clear to anyone reading the command line. ==== With gawk-devel's optional fourth parameter of split() To print from the third to the last field with [[http://cvs.savannah.gnu.org/viewvc/gawk-devel/?root=gawk|gawk-devel]]'s {{{split()}}}, when the field separator is a full regular expression: {{{ bash awk ' { nf = split($0, fld, fs_regex, delim) for (i = 3; i <= NF; ++i) printf "%s%s", fld[i], ((i < NF) ? delim[i] : "\n") }' }}} The fourth (optional) argument {{{delim}}} of {{{split()}}} is an array where {{{delim[i]}}} gets filled with the delimiter string between {{{fld[i]}}} and {{{fld[i+1]}}}. {{{fld[0]}}} is the delimiter prefix and {{{fld[nf]}}} the delimiter suffix, if the field separator is {{{" "}}} (a blank); for a regular expression field separator {{{fld[0]}}} and {{{fld[nf]}}} don't exist. ==== Function using match() and substr() This approach uses match() to get the position and length of each field separator, then substr() to either trim it from the beginning or add that field and it's succeeding separator to an output string. Afterwards, it appends the last field without the separator to the output string and returns said string. {{{ awk # usage: extract_range(string, start, stop) # extracts fields "start" through "stop" from "string", based on FS, with the # original field separators intact. returns the extracted fields. function extract_range(str, start, stop, i, re, out) { # if FS is the default, trim leading and trailing spaces from "string" and # set "re" to the appropriate regex if (FS == " ") { gsub(/^[[:space:]]+|[[:space:]]+$/, "", str); re = "[[:space:]]+"; } else { re = FS; } # remove fields 1 through start - 1 from the beginning for (i=1; i<start; i++) { if (match(str, re)) { str = substr(str, RSTART + RLENGTH); # there's no FS left, therefore the range is empty } else { return ""; } } # add fields start through stop - 1 to the output var for (i=start; i<stop; i++) { if (match(str, re)) { # append the field to the output out = out substr(str, 1, RSTART + RLENGTH - 1); # remove the field from the line str = substr(str, RSTART + RLENGTH); # no FS left, just append the rest of the line and return } else { return out str; } } # append the last field and return if (match(str, re)) { return out substr(str, 1, RSTART - 1); } else { return out str; } } # example use to print $3 through the end: awk '{print extract_range($0, 3, NF)}' }}} ==== Using index(), substr(), length() and printf() This approach finds the length of each field, starting from the end of the last field (meaning that any whitespace is included). It then uses printf to space-pad the field to that length, therefore preserving the original whitespace. It works best when the fields are separated by spaces, although it could be adapted for other field separators. Note that tabs will be treated as one character, and replaced with a single space. {{{ bash # You could hardcode the numbers, replacing "s" and "e" in the code. # If you wanted "3 to the end", as above, simply replace "e" with "NF". awk -v s="$start" -v e="$end" ' { # the ending offset of the last field, from the beginning, is stored in "prev" prev = 0; # first, just add the lengths of fields 1 through s - 1 to prev for (i=1; i<s; i++) { prev += index(substr($0, prev + 1), $i) + length($i) - 1; } # add the space between s-1 and s to prev prev += index(substr($0, prev + 1), $s) - 1; # loop over the fields we want to print for (i=s; i<=e; i++) { # get the length, from the end of the last field to the end of the current len = index(substr($0, prev + 1), $i) + length($i) - 1; # print the field, padded to that length printf("%*s%s", len, $i, i==e ? "\n" : ""); # add the length to "prev" prev += len; } }' # as a (granted, long) one-liner, printing 3 through the end awk '{p=0; for (i=1;i<3;i++) p+=index(substr($0,p+1),$i)+length($i)-1; p+=index(substr($0,p+1),$3)-1; for (i=3;i<=NF;i++) {p+=l=index(substr($0,p+1),$i)+length($i)-1; printf("%*s%s",l,$i,i==NF?"\n":"")}}' }}} ---- An advantage to this method is that you could also use it to process/change fields in a table, and keep the format pretty much the same. The following script doesn't actually remove any fields from the output, but allows you to change the second field and still pad everything the same way: {{{ awk #!/usr/bin/awk -f # store the length of each field from the end of the previous { prev = 0; for (i=1; i<=NF; i++) { lens[i] = index(substr($0, prev + 1), $i) + length($i) - 1; } } # do processing, reassignments, whatever here { $2 = "new"; } # print fields, padded appropriately { for (i=1; i<=NF; i++) { printf("%*s%s", lens[i], $i, i==NF ? "\n" : ""); } } }}} ==== Some other approaches Eric Pement put together [http://www.pement.org/awk/awktail.txt|a short list] of tactics.

Summary:

This change is a minor edit.

Username: