Question

IST 411

1. What are the advantages of using CURL for handling webbot automation?
2. What are some of the disadvantages in creating a webot to perform content gathering?
3. What networking protocols are supported with CURL?
4. What is pycurl and what can you do with pycurl?
5. Write a pycurl example that grabs some Web page HTML content.

Answer 1

1. What are the advantages of using CURL for handling web bot automation?

cURL can do all sorts of tricks in addition to merely downloading files. And cURL is very useful to test with a cookie, RESTful APIs, proxy support, user authentication, FTP upload, HTTP post, SSL connections, cookies, file transfer resume, remotely parsing web pages.

3. What network protocols are supported with CURL?

The URL syntax is protocol-dependent. eg. http{s}://www.name.com

• HTTP(HyperText Transfer Protocol)
• HTTPS(HyperText Transfer Protocol Secure)
• TFTP(Trivial File Transfer Protocol )
• FTP(File Transfer Protocol)
• FTPS(File Transfer Protocol Secure)
• TELNET9TELecommunication NETwork)
• DICT( Dictionary Server Protocol),
• IMAP(Internet Message Access Protocol), IMAPS,
• LDAP(Lightweight Directory Access Protocol ), LDAPS
• POP3(Post Office Protocol 3), POP3S,
• SFTP(Secure File Transfer Protocol)
• SMB(Server Message Block), SMBS
• SMTP(Simple Mail Transfer Protocol), SMTPS,
• RTMP(Real-Time Messaging Protocol)
• GOPHER, RTSP(Real-Time Streaming Protocol ), SCP

4. What is pycurl and what can you do with pycurl?

py+curl = python + transfer URL -- fetch URL using python programming.

pyURL is a Python interface to libcurl

USES:

• It is very fast compared to libcURL.
• supports most of the libcURL callbacks
• Share and Multi interfaces
• support SSL, authentication and proxy options
• will be very helpful for network operations

5. Write a pycurl example that grabs some Web page HTML content.

import pycurl

from StringIO import StringIO

url = 'http://www.google.com/'

data = StringIO()

c = pycurl.Curl()

c.setopt(c.URL, url)

c.setopt(c.WRITEDATA, data)

c.perform()

c.close()

value = data.getvalue()

print (value)