Addons/web/gethttp

web/gethttp - Retrieve web pages/files from the Internet using Wget or cURL.

• Provides simple interface to external utilities for retrieving URL contents from the Internet
• Uses Wget on Windows and Linux, cURL on Mac
• Write URL contents to session or file
• Ability to provide full command line options to the utilities if desired

Browse history, source and examples using github.

Verbs available

urlencode	v	Encode string as valid url
gethttp	v	Retrieve URI using Wget/cURL tools

Installation

Use JAL/Package Manager.

Load web/gethttp addon with the following line:

   load 'web/gethttp'

Check it is working OK:

   load 'web/gethttp/test/test_gethttp'
test_gethttp passed

If you are behind a proxy server you will need to configure Wget/cURL accordingly, but you have probably already done this so as to get Package Manager to work for you.

Examples of use

Retrieve Jsoftware homepage to the session:

   homepagehtml=: gethttp 'http://www.jsoftware.com'
   106 {. 306}.homepagehtml
<div id="Header">
<div id="Logo">
<a href="./zippy.htm"><img alt="JSoftware" src="./jsoftwarea.gif"></a>

Retrieve Zippy image and write it to current folder:

   'file' gethttp 'http://www.jsoftware.com/zippy.gif'
--11:28:50--  http://www.jsoftware.com/zippy.gif
           => `zippy.gif'
Resolving server.domain.com... 123.123.123.123
Connecting to server.domain.com|123.123.123.123|:8080... connected.
Proxy request sent, awaiting response... 200 OK
Length: 43,143 (42K) [image/gif]

    0K .......... .......... .......... .......... ..        100%   53.52 KB/s

11:28:51 (53.52 KB/s) - `zippy.gif' saved [43143/43143]

Retrieve Zippy image and write it to the ~temp folder as cartoon.gif:

   ('file';jpath '~temp/cartoon.gif') gethttp 'http://www.jsoftware.com/zippy.gif'
--11:30:25--  http://www.jsoftware.com/zippy.gif
           => `C:/Users/thisuser/Documents/j602-user/temp/cartoon.gif'
Resolving server.domain.com... 123.123.123.123
Connecting to server.domain.com|123.123.123.123|:8080... connected.
Proxy request sent, awaiting response... 200 OK
Length: 43,143 (42K) [image/gif]

    0K .......... .......... .......... .......... ..        100%   14.93 MB/s

11:30:25 (14.93 MB/s) - `C:/Users/thisuser/Documents/j602-user/temp/cartoon.gif' saved [43143/43143]

Use custom options for external utility.
This is a Wget example that retrieves content from stdout and saves the log to file:

   106 {. 306}. ('-O - -o ',jpath '~temp/gethttp.log') gethttp 'http://www.jsoftware.com'
<div id="Header">
<div id="Logo">
<a href="./zippy.htm"><img alt="JSoftware" src="./jsoftwarea.gif"></a>

See test_gethttp.ijs for more usage examples.

Documentation

gethttp

gethttp (v) Retrieve URI using Wget/cURL tools
form: [option] gethttp uri
result: depends on options, Default is URI contents
y is: URI to retrieve
x is: Optional retrieval options. One of:
      'stdout' (Default)
      'help'
      'file' or ('file';jpath '~temp/myfile.htm')
      Anything else is assumed to be a valid Wget/cURL option string
eg: 'file' gethttp 'http://www.jsoftware.com'
eg: ('-o - --stderr ',jpath '~temp/gethttp.log') gethttp 'http://www.jsoftware.com'

Known Issues

When retrieving a binary file to the session on Windows for example: myimg=: gethttp 'http://www.mydomain.com/myimage.jpg' , any LF characters in the file will be replaced by CRLF in the session output.

A workaround is to retrieve the binary file and write to disk for example: 'file' gethttp 'http://www.mydomain.com/myimage.jpg' and then read the file from disk into the session.

Author

Ric Sherlock

See Also

• Scripts/HTTP Get is a J sockets solution for retrieving files from the Internet.