Sed Usage in Shell Programming lufy December 07, 2019 <p>from Advanced Bash-Scripting Guide:</p> <p><a href="http://tldp.org/LDP/abs/html/x23170.html" target="_blank">http://tldp.org/LDP/abs/html/x23170.html</a></p> <h1 class="SECT1">Sed</h1> <p><em class="FIRSTTERM">Sed</em> is a non-interactive <a name="AEN23174" href="http://tldp.org/LDP/abs/html/x23170.html#FTN.AEN23174"><span class="footnote">[1]</span></a> <strong class="COMMAND">s</strong>tream <strong class="COMMAND">ed</strong>itor. It receives text input, whether from <tt class="FILENAME">stdin</tt> or from a file, performs certain operations on specified lines of the input, one line at a time, then outputs the result to <tt class="FILENAME">stdout</tt> or to a file. Within a shell script, <em class="FIRSTTERM">sed</em> is usually one of several tool components in a pipe.</p> <p><em class="FIRSTTERM">Sed</em> determines which lines of its input that it will operate on from the <em class="FIRSTTERM">address range</em> passed to it. <a name="AEN23185" href="http://tldp.org/LDP/abs/html/x23170.html#FTN.AEN23185"><span class="footnote">[2]</span></a> Specify this address range either by line number or by a pattern to match. For example, <tt class="REPLACEABLE"><em>3d</em></tt> signals <em class="FIRSTTERM">sed</em> to delete line 3 of the input, and <tt class="REPLACEABLE"><em>/Windows/d</em></tt> tells sed that you want every line of the input containing a match to <span class="QUOTE">"Windows"</span> deleted.</p> <p>Of all the operations in the <em class="FIRSTTERM">sed</em> toolkit, we will focus primarily on the three most commonly used ones. These are <strong class="COMMAND">p</strong>rinting (to <tt class="FILENAME">stdout</tt>), <strong class="COMMAND">d</strong>eletion, and <strong class="COMMAND">s</strong>ubstitution.</p> <p><a name="SEDBASICTABLE"></a></p> <div class="TABLE"><a name="AEN23200"></a> <p><strong>Table C-1. Basic sed operators</strong></p> <table class="CALSTABLE" border="1"> <thead> <tr><th align="LEFT" valign="TOP">Operator</th><th align="LEFT" valign="TOP">Name</th><th align="LEFT" valign="TOP">Effect</th></tr> </thead> <tbody> <tr> <td align="LEFT" valign="TOP"><tt class="OPTION">[address-range]/p</tt></td> <td align="LEFT" valign="TOP">print</td> <td align="LEFT" valign="TOP">Print [specified address range]</td> </tr> <tr> <td align="LEFT" valign="TOP"><tt class="OPTION">[address-range]/d</tt></td> <td align="LEFT" valign="TOP">delete</td> <td align="LEFT" valign="TOP">Delete [specified address range]</td> </tr> <tr> <td align="LEFT" valign="TOP"><tt class="OPTION">s/pattern1/pattern2/</tt></td> <td align="LEFT" valign="TOP">substitute</td> <td align="LEFT" valign="TOP">Substitute pattern2 for first instance of pattern1 in a line</td> </tr> <tr> <td align="LEFT" valign="TOP"><tt class="OPTION">[address-range]/s/pattern1/pattern2/</tt></td> <td align="LEFT" valign="TOP">substitute</td> <td align="LEFT" valign="TOP">Substitute pattern2 for first instance of pattern1 in a line, over <tt class="REPLACEABLE"><em>address-range</em></tt></td> </tr> <tr> <td align="LEFT" valign="TOP"><tt class="OPTION">[address-range]/y/pattern1/pattern2/</tt></td> <td align="LEFT" valign="TOP">transform</td> <td align="LEFT" valign="TOP">replace any character in pattern1 with the corresponding character in pattern2, over <tt class="REPLACEABLE"><em>address-range</em></tt> (equivalent of <strong class="COMMAND">tr</strong>)</td> </tr> <tr> <td align="LEFT" valign="TOP"><tt class="OPTION">[address] i pattern Filename</tt></td> <td align="LEFT" valign="TOP">insert</td> <td align="LEFT" valign="TOP">Insert pattern at address indicated in file Filename. Usually used with <tt class="OPTION">-i</tt> <tt class="REPLACEABLE"><em>in-place</em></tt> option.</td> </tr> <tr> <td align="LEFT" valign="TOP"><tt class="OPTION">g</tt></td> <td align="LEFT" valign="TOP">global</td> <td align="LEFT" valign="TOP">Operate on <em>every</em> pattern match within each matched line of input</td> </tr> </tbody> </table> </div> <div class="NOTE"> <table class="NOTE" style="width: 100%;" border="0"> <tbody> <tr> <td align="CENTER" valign="TOP" width="25"><img src="http://tldp.org/LDP/abs/images/note.gif" alt="Note" hspace="5" /></td> <td align="LEFT" valign="TOP"> <p>Unless the <tt class="OPTION">g</tt> (<em class="FIRSTTERM">global</em>) operator is appended to a <em class="FIRSTTERM">substitute</em> command, the substitution operates only on the <em>first</em> instance of a pattern match within each line.</p> </td> </tr> </tbody> </table> </div> <p>From the command-line and in a shell script, a sed operation may require quoting and certain options.</p> <table style="width: 100%;" border="0" bgcolor="#E0E0E0"> <tbody> <tr> <td> <pre class="PROGRAMLISTING">sed -e '/^$/d' $filename # The -e option causes the next string to be interpreted as an editing instruction. # (If passing only a single instruction to sed, the "-e" is optional.) # The "strong" quotes ('') protect the RE characters in the instruction #+ from reinterpretation as special characters by the body of the script. # (This reserves RE expansion of the instruction for sed.) # # Operates on the text contained in file $filename.</pre> </td> </tr> </tbody> </table> <p>In certain cases, a <em class="FIRSTTERM">sed</em> editing command will not work with single quotes.</p> <table style="width: 100%;" border="0" bgcolor="#E0E0E0"> <tbody> <tr> <td> <pre class="PROGRAMLISTING">filename=file1.txt pattern=BEGIN sed "/^$pattern/d" "$filename" # Works as specified. # sed '/^$pattern/d' "$filename" has unexpected results. # In this instance, with strong quoting (' ... '), #+ "$pattern" will not expand to "BEGIN".</pre> </td> </tr> </tbody> </table> <div class="NOTE"> <table class="NOTE" style="width: 100%;" border="0"> <tbody> <tr> <td align="CENTER" valign="TOP" width="25"><img src="http://tldp.org/LDP/abs/images/note.gif" alt="Note" hspace="5" /></td> <td align="LEFT" valign="TOP"> <p><em class="FIRSTTERM">Sed</em> uses the <tt class="OPTION">-e</tt> option to specify that the following string is an instruction or set of instructions. If there is only a single instruction contained in the string, then this may be omitted.</p> </td> </tr> </tbody> </table> </div> <table style="width: 100%;" border="0" bgcolor="#E0E0E0"> <tbody> <tr> <td> <pre class="PROGRAMLISTING">sed -n '/xzy/p' $filename # The -n option tells sed to print only those lines matching the pattern. # Otherwise all input lines would print. # The -e option not necessary here since there is only a single editing instruction.</pre> </td> </tr> </tbody> </table> <p><a name="SEDOPTABLE"></a></p> <div class="TABLE"><a name="AEN23271"></a> <p><strong>Table C-2. Examples of sed operators</strong></p> <table class="CALSTABLE" border="1"> <thead> <tr><th align="LEFT" valign="TOP">Notation</th><th align="LEFT" valign="TOP">Effect</th></tr> </thead> <tbody> <tr> <td align="LEFT" valign="TOP"><tt class="OPTION">8d</tt></td> <td align="LEFT" valign="TOP">Delete 8th line of input.</td> </tr> <tr> <td align="LEFT" valign="TOP"><tt class="OPTION">/^$/d</tt></td> <td align="LEFT" valign="TOP">Delete all blank lines.</td> </tr> <tr> <td align="LEFT" valign="TOP"><tt class="OPTION">1,/^$/d</tt></td> <td align="LEFT" valign="TOP">Delete from beginning of input up to, and including first blank line.</td> </tr> <tr> <td align="LEFT" valign="TOP"><tt class="OPTION">/Jones/p</tt></td> <td align="LEFT" valign="TOP">Print only lines containing <span class="QUOTE">"Jones"</span> (with <span class="TOKEN">-n</span> option).</td> </tr> <tr> <td align="LEFT" valign="TOP"><tt class="OPTION">s/Windows/Linux/</tt></td> <td align="LEFT" valign="TOP">Substitute <span class="QUOTE">"Linux"</span> for first instance of <span class="QUOTE">"Windows"</span> found in each input line.</td> </tr> <tr> <td align="LEFT" valign="TOP"><tt class="OPTION">s/BSOD/stability/g</tt></td> <td align="LEFT" valign="TOP">Substitute <span class="QUOTE">"stability"</span> for every instance of <span class="QUOTE">"BSOD"</span> found in each input line.</td> </tr> <tr> <td align="LEFT" valign="TOP"><tt class="OPTION">s/ *$//</tt></td> <td align="LEFT" valign="TOP">Delete all spaces at the end of every line.</td> </tr> <tr> <td align="LEFT" valign="TOP"><tt class="OPTION">s/00*/0/g</tt></td> <td align="LEFT" valign="TOP">Compress all consecutive sequences of zeroes into a single zero.</td> </tr> <tr> <td align="LEFT" valign="TOP"><tt class="OPTION">echo "Working on it." | sed -e '1i How far are you along?'</tt></td> <td align="LEFT" valign="TOP">Prints "How far are you along?" as first line, "Working on it" as second.</td> </tr> <tr> <td align="LEFT" valign="TOP"><tt class="OPTION">5i 'Linux is great.' file.txt</tt></td> <td align="LEFT" valign="TOP">Inserts 'Linux is great.' at line 5 of the file file.txt.</td> </tr> <tr> <td align="LEFT" valign="TOP"><tt class="OPTION">/GUI/d</tt></td> <td align="LEFT" valign="TOP">Delete all lines containing <span class="QUOTE">"GUI"</span>.</td> </tr> <tr> <td align="LEFT" valign="TOP"><tt class="OPTION">s/GUI//g</tt></td> <td align="LEFT" valign="TOP">Delete all instances of <span class="QUOTE">"GUI"</span>, leaving the remainder of each line intact.</td> </tr> </tbody> </table> </div> <p>Substituting a zero-length string for another is equivalent to deleting that string within a line of input. This leaves the remainder of the line intact. Applying <tt class="USERINPUT"><strong>s/GUI//</strong></tt> to the line</p> <table style="width: 100%;" border="1" bgcolor="#E0E0E0"> <tbody> <tr> <td> <pre class="SCREEN"><tt class="USERINPUT"><strong>The most important parts of any application are its GUI and sound effects</strong></tt></pre> </td> </tr> </tbody> </table> <p>results in</p> <table style="width: 100%;" border="1" bgcolor="#E0E0E0"> <tbody> <tr> <td> <pre class="SCREEN"><tt class="COMPUTEROUTPUT">The most important parts of any application are its and sound effects</tt></pre> </td> </tr> </tbody> </table> <p>A backslash forces the <strong class="COMMAND">sed</strong> replacement command to continue on to the next line. This has the effect of using the <em class="FIRSTTERM">newline</em> at the end of the first line as the <em class="FIRSTTERM">replacement string</em>.</p> <table style="width: 100%;" border="0" bgcolor="#E0E0E0"> <tbody> <tr> <td> <pre class="PROGRAMLISTING">s/^ */\ /g</pre> </td> </tr> </tbody> </table> <p>This substitution replaces line-beginning spaces with a newline. The net result is to replace paragraph indents with a blank line between paragraphs.</p> <p>An address range followed by one or more operations may require open and closed curly brackets, with appropriate newlines.</p> <table style="width: 100%;" border="0" bgcolor="#E0E0E0"> <tbody> <tr> <td> <pre class="PROGRAMLISTING">/[0-9A-Za-z]/,/^$/{ /^$/d }</pre> </td> </tr> </tbody> </table> <p>This deletes only the first of each set of consecutive blank lines. That might be useful for single-spacing a text file, but retaining the blank line(s) between paragraphs.</p> <div class="NOTE"> <table class="NOTE" style="width: 100%;" border="0"> <tbody> <tr> <td align="CENTER" valign="TOP" width="25"><img src="http://tldp.org/LDP/abs/images/note.gif" alt="Note" hspace="5" /></td> <td align="LEFT" valign="TOP"> <p>The usual delimiter that <em class="FIRSTTERM">sed</em> uses is <span class="TOKEN">/</span>. However, <em>sed</em> allows other delimiters, such as <span class="TOKEN">%</span>. This is useful when <span class="TOKEN">/</span> is part of a replacement string, as in a file pathname. See <a href="http://tldp.org/LDP/abs/html/loops1.html#FINDSTRING">Example 11-10</a> and <a href="http://tldp.org/LDP/abs/html/filearchiv.html#STRIPC">Example 16-32</a>.</p> </td> </tr> </tbody> </table> </div> <p><a name="DOUBLESPACE"></a></p> <div class="TIP"> <table class="TIP" style="width: 100%;" border="0"> <tbody> <tr> <td align="CENTER" valign="TOP" width="25"><img src="http://tldp.org/LDP/abs/images/tip.gif" alt="Tip" hspace="5" /></td> <td align="LEFT" valign="TOP"> <p>A quick way to double-space a text file is <tt class="USERINPUT"><strong>sed G filename</strong></tt>.</p> </td> </tr> </tbody> </table> </div> <p>For illustrative examples of sed within shell scripts, see:</p> <ol type="1"> <li> <p><a href="http://tldp.org/LDP/abs/html/wrapper.html#EX3">Example 36-1</a></p> </li> <li> <p><a href="http://tldp.org/LDP/abs/html/wrapper.html#EX4">Example 36-2</a></p> </li> <li> <p><a href="http://tldp.org/LDP/abs/html/moreadv.html#EX57">Example 16-3</a></p> </li> <li> <p><a href="http://tldp.org/LDP/abs/html/contributed-scripts.html#RN">Example A-2</a></p> </li> <li> <p><a href="http://tldp.org/LDP/abs/html/textproc.html#GRP">Example 16-17</a></p> </li> <li> <p><a href="http://tldp.org/LDP/abs/html/textproc.html#COL">Example 16-27</a></p> </li> <li> <p><a href="http://tldp.org/LDP/abs/html/contributed-scripts.html#BEHEAD">Example A-12</a></p> </li> <li> <p><a href="http://tldp.org/LDP/abs/html/contributed-scripts.html#TREE">Example A-16</a></p> </li> <li> <p><a href="http://tldp.org/LDP/abs/html/contributed-scripts.html#TREE2">Example A-17</a></p> </li> <li> <p><a href="http://tldp.org/LDP/abs/html/filearchiv.html#STRIPC">Example 16-32</a></p> </li> <li> <p><a href="http://tldp.org/LDP/abs/html/loops1.html#FINDSTRING">Example 11-10</a></p> </li> <li> <p><a href="http://tldp.org/LDP/abs/html/mathc.html#BASE">Example 16-48</a></p> </li> <li> <p><a href="http://tldp.org/LDP/abs/html/contributed-scripts.html#MAILFORMAT">Example A-1</a></p> </li> <li> <p><a href="http://tldp.org/LDP/abs/html/textproc.html#RND">Example 16-14</a></p> </li> <li> <p><a href="http://tldp.org/LDP/abs/html/textproc.html#WF">Example 16-12</a></p> </li> <li> <p><a href="http://tldp.org/LDP/abs/html/contributed-scripts.html#LIFESLOW">Example A-10</a></p> </li> <li> <p><a href="http://tldp.org/LDP/abs/html/here-docs.html#SELFDOCUMENT">Example 19-12</a></p> </li> <li> <p><a href="http://tldp.org/LDP/abs/html/textproc.html#DICTLOOKUP">Example 16-19</a></p> </li> <li> <p><a href="http://tldp.org/LDP/abs/html/contributed-scripts.html#WHX">Example A-29</a></p> </li> <li> <p><a href="http://tldp.org/LDP/abs/html/contributed-scripts.html#BASHPODDER">Example A-31</a></p> </li> <li> <p><a href="http://tldp.org/LDP/abs/html/contributed-scripts.html#TOHTML">Example A-24</a></p> </li> <li> <p><a href="http://tldp.org/LDP/abs/html/contributed-scripts.html#STOPWATCH">Example A-43</a></p> </li> <li> <p><a href="http://tldp.org/LDP/abs/html/contributed-scripts.html#SEDAPPEND">Example A-55</a></p> </li> </ol> <p>For a more extensive treatment of <em class="FIRSTTERM">sed</em>, refer to the <a href="http://tldp.org/LDP/abs/html/biblio.html#DGSEDREF">pertinent references</a> in the <a href="http://tldp.org/LDP/abs/html/biblio.html"><em>Bibliography</em></a>.</p> <h3 class="FOOTNOTES">Notes</h3> <table class="FOOTNOTES" style="width: 100%;" border="0"> <tbody> <tr> <td align="LEFT" valign="TOP" width="5%"><a name="FTN.AEN23174" href="http://tldp.org/LDP/abs/html/x23170.html#AEN23174"><span class="footnote">[1]</span></a></td> <td align="LEFT" valign="TOP" width="95%"> <p><em class="FIRSTTERM">Sed</em> executes without user intervention.</p> </td> </tr> <tr> <td align="LEFT" valign="TOP" width="5%"><a name="FTN.AEN23185" href="http://tldp.org/LDP/abs/html/x23170.html#AEN23185"><span class="footnote">[2]</span></a></td> <td align="LEFT" valign="TOP" width="95%"> <p>If no address range is specified, the default is <em>all</em> lines.</p> </td> </tr> </tbody> </table>
Comments (0)
Leave a Comment
No comments yet. Be the first to comment!