SEO

Hi all.

I'm attempting to script an FTP transfer on my UnixWare 7.1.4 server. I've
written numerous FTP transfer scripts before, but haven't really approached
it this way.

What I'd like to do is basically retrieve some files from a remote system,
then move them at the remote end into an archive folder. Sounds pretty
simple.

In my script, I'm trying to write a FTP command file on the fly to redirect
into my FTP command as follows:-

===========
:
ftphost=ftp.XXXXXX.com
ftpuser=XXXXXX
ftppass=XXXXXX
ftpdir="/"
ftparcdir="/arcdir"
work=/tmp/xfer.$$
#
# get list of files waiting
ftp -g -n ${ftphost} > ${work} 2>&1 << !
user ${ftpuser} ${ftppass}
cd ${ftpdir}
dir
bye
!

# process file list into ftp commands
exec 3< ${work}
while read line <&3
do
filename=`echo ${line} | awk '{print $4}'`
fileext=`echo ${filename} | awk -F\. '{print $2}' | dd conv=ucase
2>/dev/null`
if [ "${fileext}" = "XML" ]
then
echo "get ${filename}" >> ${work}.2
echo "ren ${filename} ${ftparcdir}/${filename}" >> ${work}.3
fi
done

# write ftp script file to retrieve all XML files
#echo "OPEN ${ftphost}" > /tmp/ftp.$$
echo "USER ${ftpuser} ${ftppass}" >> /tmp/ftp.$$
echo "binary" >> /tmp/ftp.$$
echo "cd ${ftpdir}" >> /tmp/ftp.$$
cat ${work}.2 >> /tmp/ftp.$$
cat ${work}.3 >> /tmp/ftp.$$
echo "bye" >> /tmp/ftp.$$

# run script to transfer all XML files and archive them
ftp -i -g -n ${ftphost} < /tmp/ftp.$$ > /tmp/ftp$$.log 2>&1
rm -f ${work} ${work}.2 ${work}.3

cat /tmp/ftp$$.log
rm -f /tmp/ftp$$.log
rm -f /tmp/ftp.$$

exit 0

===========

The script runs and creates my command file, but I just cannot get it to
accept the commands from the temporary file. I have removed the redirect to
the ftp$$.log file, and stripped out the relevant commands and run them
directly from the command line.

The output I get (either on the screen, or in the ftp$$.log file) is:-

?Invalid command
?Invalid command
Not connected.
Not connected.
Not connected.
Not connected.
Not connected.
Not connected.


This is the method for ftp which I have found on Google, but cannot
understand why it isn't working on my Unixware system. I get exactly the
same results on an Openserver 5.0.6 system too.

Any ideas what I'm doing wrong here?


I suppose I could also do a "mget *.xml" in this instance, but not knowing
the case of the files (it could be another Unix or Linux system at the other
end - I don't know), this may not work.



Many thanks,

Julian.

ThreeStar wrote:

> Sounds like a problem with the command file (/tmp/ftp.$$). Post
> that. From the "Not connected" errors I'm guessing it's a problem
> with establishing user credentials. Are you able to FTP interactively
> to the host from this box?
>
> BTW if this script is designed to run in a particular context I'd
> use .netrc files to contain the user name and password instead of
> echoing them in the script. It's somewhat more secure, makes the
> password easier to change, and simplifies your script.
>
> Better yet is SCP, but you didn't ask about that.
>
> R Robert
> *** Software

SCP is *AWFUL* for this sort of thing. It mishandles symlinks, and SCP or SFTP
access provides user access to the rest of the server's file system, with the
user's normal privileges. That presents a real security issue.

I'm a big proponent of WebDAV over HTTPS fur just this sort of thing, and for
rsync over SSH with a restricted validation script as needed.

On Tue, Apr 22, 2008, Nico Kadel-Garcia wrote:
>ThreeStar wrote:
>
>> Sounds like a problem with the command file (/tmp/ftp.$$). Post
>> that. From the "Not connected" errors I'm guessing it's a problem
>> with establishing user credentials. Are you able to FTP interactively
>> to the host from this box?
>>
>> BTW if this script is designed to run in a particular context I'd
>> use .netrc files to contain the user name and password instead of
>> echoing them in the script. It's somewhat more secure, makes the
>> password easier to change, and simplifies your script.
>>
>> Better yet is SCP, but you didn't ask about that.
>>
>> R Robert
>> *** Software
>
>SCP is *AWFUL* for this sort of thing. It mishandles symlinks, and SCP or SFTP
>access provides user access to the rest of the server's file system, with the
>user's normal privileges. That presents a real security issue.
>
>I'm a big proponent of WebDAV over HTTPS fur just this sort of thing, and for
>rsync over SSH with a restricted validation script as needed.

Depending on the nature of the data, we often use rsync's modules
to handle things like this, restricting access by IP address.
Doing it that way does not require things like ssh identities with
empty pass phrases or other potential security issues.

Bill
--
INTERNET: bill@celestial.com Bill Campbell; Celestial Software LLC
URL: http://www.celestial.com/ PO Box 820; 6641 E. Mercer Way
Voice: (206) 236-1676 Mercer Island, WA 98040-0820
Fax: (206) 232-9186

Each individual of the society has a right to be protected in the
enjoyment of his life, liberty, and property no part of the property
of any individual can, with justice, be taken from him, or applied to
public uses, without his own consent -- John Adams

On Apr 22, 2:34 am, "Julian Brett" <nos...@nospamhere.com> wrote:
> Hi all.
>
> I'm attempting to script an FTP transfer on my UnixWare 7.1.4 server. I've
> written numerous FTP transfer scripts before, but haven't really approached
> it this way.
>
> What I'd like to do is basically retrieve some files from a remote system,
> then move them at the remote end into an archive folder. Sounds pretty
> simple.
>
> In my script, I'm trying to write a FTP command file on the fly to redirect
> into my FTP command as follows:-
>
> ===========
> :
> ftphost=ftp.XXXXXX.com
> ftpuser=XXXXXX
> ftppass=XXXXXX
> ftpdir="/"
> ftparcdir="/arcdir"
> work=/tmp/xfer.$$
> #
> # get list of files waiting
> ftp -g -n ${ftphost} > ${work} 2>&1 << !
> user ${ftpuser} ${ftppass}
> cd ${ftpdir}
> dir
> bye
> !
>
> # process file list into ftp commands
> exec 3< ${work}
> while read line <&3
> do
> filename=`echo ${line} | awk '{print $4}'`
> fileext=`echo ${filename} | awk -F\. '{print $2}' | dd conv=ucase
> 2>/dev/null`
> if [ "${fileext}" = "XML" ]
> then
> echo "get ${filename}" >> ${work}.2
> echo "ren ${filename} ${ftparcdir}/${filename}" >> ${work}.3
> fi
> done
>
> # write ftp script file to retrieve all XML files
> #echo "OPEN ${ftphost}" > /tmp/ftp.$$
> echo "USER ${ftpuser} ${ftppass}" >> /tmp/ftp.$$
> echo "binary" >> /tmp/ftp.$$
> echo "cd ${ftpdir}" >> /tmp/ftp.$$
> cat ${work}.2 >> /tmp/ftp.$$
> cat ${work}.3 >> /tmp/ftp.$$
> echo "bye" >> /tmp/ftp.$$
>
> # run script to transfer all XML files and archive them
> ftp -i -g -n ${ftphost} < /tmp/ftp.$$ > /tmp/ftp$$.log 2>&1
> rm -f ${work} ${work}.2 ${work}.3
>
> cat /tmp/ftp$$.log
> rm -f /tmp/ftp$$.log
> rm -f /tmp/ftp.$$
>
> exit 0
>
> ===========
>
> The script runs and creates my command file, but I just cannot get it to
> accept the commands from the temporary file. I have removed the redirect to
> the ftp$$.log file, and stripped out the relevant commands and run them
> directly from the command line.
>
> The output I get (either on the screen, or in the ftp$$.log file) is:-
>
> ?Invalid command
> ?Invalid command
> Not connected.
> Not connected.
> Not connected.
> Not connected.
> Not connected.
> Not connected.
>
> This is the method for ftp which I have found on Google, but cannot
> understand why it isn't working on my Unixware system. I get exactly the
> same results on an Openserver 5.0.6 system too.
>
> Any ideas what I'm doing wrong here?
>
> I suppose I could also do a "mget *.xml" in this instance, but not knowing
> the case of the files (it could be another Unix or Linux system at the other
> end - I don't know), this may not work.
>
> Many thanks,
>
> Julian.

Sounds like a problem with the command file (/tmp/ftp.$$). Post
that. From the "Not connected" errors I'm guessing it's a problem
with establishing user credentials. Are you able to FTP interactively
to the host from this box?

BTW if this script is designed to run in a particular context I'd
use .netrc files to contain the user name and password instead of
echoing them in the script. It's somewhat more secure, makes the
password easier to change, and simplifies your script.

Better yet is SCP, but you didn't ask about that.

R Robert
*** Software

Rather then specify the username and password in your script, I would
suggest you put the ftp destination name/IP address, user name and password
details in to a .netrc file in the home directory of the user name that is
going to run the ftp script, so....

If you are running this from root, the .netrc will be created in /

The format of .netrc would be:

machine name/IP address login username password password

set chmod to 600 for .netrc. Then run your script minus the ftpuser and
ftppass variables

Dave

"Julian Brett" <nospam@nospamhere.com> wrote in message
news:fukbc8$mej$1$8300dec7@news.demon.co.uk...
> Hi all.
>
> I'm attempting to script an FTP transfer on my UnixWare 7.1.4 server.
> I've written numerous FTP transfer scripts before, but haven't really
> approached it this way.
>
> What I'd like to do is basically retrieve some files from a remote system,
> then move them at the remote end into an archive folder. Sounds pretty
> simple.
>
> In my script, I'm trying to write a FTP command file on the fly to
> redirect into my FTP command as follows:-
>
> ===========
> :
> ftphost=ftp.XXXXXX.com
> ftpuser=XXXXXX
> ftppass=XXXXXX
> ftpdir="/"
> ftparcdir="/arcdir"
> work=/tmp/xfer.$$
> #
> # get list of files waiting
> ftp -g -n ${ftphost} > ${work} 2>&1 << !
> user ${ftpuser} ${ftppass}
> cd ${ftpdir}
> dir
> bye
> !
>
> # process file list into ftp commands
> exec 3< ${work}
> while read line <&3
> do
> filename=`echo ${line} | awk '{print $4}'`
> fileext=`echo ${filename} | awk -F\. '{print $2}' | dd conv=ucase
> 2>/dev/null`
> if [ "${fileext}" = "XML" ]
> then
> echo "get ${filename}" >> ${work}.2
> echo "ren ${filename} ${ftparcdir}/${filename}" >> ${work}.3
> fi
> done
>
> # write ftp script file to retrieve all XML files
> #echo "OPEN ${ftphost}" > /tmp/ftp.$$
> echo "USER ${ftpuser} ${ftppass}" >> /tmp/ftp.$$
> echo "binary" >> /tmp/ftp.$$
> echo "cd ${ftpdir}" >> /tmp/ftp.$$
> cat ${work}.2 >> /tmp/ftp.$$
> cat ${work}.3 >> /tmp/ftp.$$
> echo "bye" >> /tmp/ftp.$$
>
> # run script to transfer all XML files and archive them
> ftp -i -g -n ${ftphost} < /tmp/ftp.$$ > /tmp/ftp$$.log 2>&1
> rm -f ${work} ${work}.2 ${work}.3
>
> cat /tmp/ftp$$.log
> rm -f /tmp/ftp$$.log
> rm -f /tmp/ftp.$$
>
> exit 0
>
> ===========
>
> The script runs and creates my command file, but I just cannot get it to
> accept the commands from the temporary file. I have removed the redirect
> to the ftp$$.log file, and stripped out the relevant commands and run them
> directly from the command line.
>
> The output I get (either on the screen, or in the ftp$$.log file) is:-
>
> ?Invalid command
> ?Invalid command
> Not connected.
> Not connected.
> Not connected.
> Not connected.
> Not connected.
> Not connected.
>
>
> This is the method for ftp which I have found on Google, but cannot
> understand why it isn't working on my Unixware system. I get exactly the
> same results on an Openserver 5.0.6 system too.
>
> Any ideas what I'm doing wrong here?
>
>
> I suppose I could also do a "mget *.xml" in this instance, but not knowing
> the case of the files (it could be another Unix or Linux system at the
> other end - I don't know), this may not work.
>
>
>
> Many thanks,
>
> Julian.
>

On Apr 22, 5:34 am, "Julian Brett" <nos...@nospamhere.com> wrote:
> Hi all.
>
> I'm attempting to script an FTP transfer on my UnixWare 7.1.4 server. I've
> written numerous FTP transfer scripts before, but haven't really approached
> it this way.


Don't do this. There are better ways to transfer files. If it must
be ftp, use Perl:
http://aplawrence.com/Unixart/perlnetftp.html

Much easier, much more control, much more reliable.

On Apr 22, 8:06 am, Nico Kadel-Garcia <nka...@gmail.com> wrote:
> ThreeStar wrote:
> > Sounds like a problem with the command file (/tmp/ftp.$$). Post
> > that. From the "Not connected" errors I'm guessing it's a problem
> > with establishing user credentials. Are you able to FTP interactively
> > to the host from this box?
>
> > BTW if this script is designed to run in a particular context I'd
> > use .netrc files to contain the user name and password instead of
> > echoing them in the script. It's somewhat more secure, makes the
> > password easier to change, and simplifies your script.
>
> > Better yet is SCP, but you didn't ask about that.
>
> > R Robert
> > *** Software
>
> SCP is *AWFUL* for this sort of thing. It mishandles symlinks, and SCP or SFTP
> access provides user access to the rest of the server's file system, with the
> user's normal privileges. That presents a real security issue.
>
> I'm a big proponent of WebDAV over HTTPS fur just this sort of thing, and for
> rsync over SSH with a restricted validation script as needed.

AFAIK the only issue is that SCP copies symbolic links as files, which
isn't necessarily a bad thing.

I assume one would use SCP keys only for a user with restricted
permissions. So it doesn't necessarily follow that SCP exposes the
whole server.

Better security is better, even if it's not the best security
(whatever that means). SCP fits the simple scenario the poster laid
out, and avoids sending login credentials over the Internet in clear
text like FTP does. WebDAV and the other proposed solutions may be
better yet, although we don't know enough about the poster's equipment
or requirements or abilities to say for sure.

--RLR

On Tue, 22 Apr 2008, ThreeStar wrote:

>
> Better security is better, even if it's not the best security
> (whatever that means). SCP fits the simple scenario the poster laid
> out, and avoids sending login credentials over the Internet in clear
> text like FTP does.

There are FTP implementations available that support encryption. They
won't work through many firewalls, but as long as only one end has an
external firewall, they can be used.

On Tue, 22 Apr 2008, Joe Dunning wrote:

> There are FTP implementations available that support encryption. They
> won't work through many firewalls, but as long as only one end has an
> external firewall, they can be used.

Are you referring to FTP over SSL, sometimes known as FTPS? If not, what?

Regards,
.....Bob Rasmussen, President, Rasmussen Software, Inc.

personal e-mail: ras@anzio.com
company e-mail: rsi@anzio.com
voice: (US) 503-624-0360 (9:00-6:00 Pacific Time)
fax: (US) 503-624-0760
web: http://www.anzio.com
street address: Rasmussen Software, Inc.
10240 SW Nimbus, Suite L9
Portland, OR 97223 USA

In article <fukbc8$mej$1$8300dec7@news.demon.co.uk>, Julian Brett wrote:
> Hi all.
>
> I'm attempting to script an FTP transfer on my UnixWare 7.1.4 server. I've
> written numerous FTP transfer scripts before, but haven't really approached
> it this way.



This sort of thing in a script file has always worked for me
under various unixen:



cd_dir="somedir"
file="somefile.txt"
user="myusername"
host="myhost.com"
echo -n "Password for ${user} at ${host}: "
read pass

echo "
open ${host}
user ${user} ${pass}
#passive
bin
hash
prompt
cd ${cd_dir}
pwd
ls
chmod 700 *
mdele *
put ${file}
chmod 000 *
ls
" | ftp -n