Is anyone aware of any way of compressing the data using IConnect or
any middleware?

In our current environment, we have quite a few clients connected to
our 9.4 IDS via WAN.
The windows GUI client application is written in MFC which uses a DLL
that connects to the server using ESQL/C. The ESQL/C uses IConnect
version 2.81tc3.

The performance on the server is sufficient (CPU is usually 90% idle).
The performance for the LAN clients is sufficient; however, the
performance for the remote clients is not.

Our main MFC developer has performed experiments by writing his own
middle layer with compression instead of connecting using IConnect.
These experiments show that IConnect is a major bottleneck. While the
native speed of the WAN links is indeed slower than the LAN, the
experiements got the WAN performance level up to an accetptable level.
However, this was only an experiment, and not pursued to develop a full
solution due to the complexity and cost of the implementation. We've
then talked to IBM/Informix but it did not go anywhere. The primary
suggestion we got is to move from HDR to ER but the complication and
the cost of ER is not preferrable at this point.

Our version of ideal solution is that the IConnect layer can be
optimized, potentially with built-in compression. However, it seems
compression is available but only among the servers participated in the
ER. Why isn't it available to the client connection? has anybody knew
or written any middle layer for compression to deal with the WAN
traffic? Any suggestion is appreciated.

Thank you.

-Vivien Kobayashi

Vivien Kobayashi wrote:
> Is anyone aware of any way of compressing the data using IConnect or
> any middleware?
>
<snip>
>
> Our version of ideal solution is that the IConnect layer can be
> optimized, potentially with built-in compression. However, it seems
> compression is available but only among the servers participated in the
> ER. Why isn't it available to the client connection? has anybody knew
> or written any middle layer for compression to deal with the WAN
> traffic? Any suggestion is appreciated.

Well - you COULD write a network listener in <fill in language of your
choice> to listen on the client machine and compress data and connect
over the network to it's twin that uncompresses at the far end and
passes onto IDS.

The Stevens volume 1 of Network Programming will have client server
examples and it will be trivial to pipe IO through <fill in compression
library of your choice>

Alternatively you could hack the source of netcat to compress<->decompress

Then your client connect to 'localhost' or some other server on the lan
and then the wan connection is compressed.

I would offer to quote you for the work but that would be bad netiquette!


--
Clive Eisen
CTO Hildebrand Group

Vivien Kobayashi wrote:
> Is anyone aware of any way of compressing the data using IConnect or
> any middleware?
>
>
> Our main MFC developer has performed experiments by writing his own
> middle layer with compression instead of connecting using IConnect.
> These experiments show that IConnect is a major bottleneck. While the
> native speed of the WAN links is indeed slower than the LAN, the
> experiements got the WAN performance level up to an accetptable level.
> However, this was only an experiment, and not pursued to develop a full
> solution due to the complexity and cost of the implementation. We've
> then talked to IBM/Informix but it did not go anywhere. The primary
> suggestion we got is to move from HDR to ER but the complication and
> the cost of ER is not preferrable at this point.

I don't understand. IConnect is client/server connections. ER/HDR is
server-to-server.

Compression is useful mainly when the cost of compression is low
compared to the saved cost of network transmission. We put it in for ER
because with ER we are sending a lot of data to a lot of servers. Since
the transmission is going to be large blocks (complete transactions) and
we knew from testing that the savings would be significant, we added it
for ER. We didn't think that we would get as much savings with HDR
because we use logging compression and didn't think that we'd get good
savings.

For client server connections, the messages tend to be rather small, so
the benefit would be even less.

>
> Our version of ideal solution is that the IConnect layer can be
> optimized, potentially with built-in compression. However, it seems
> compression is available but only among the servers participated in the
> ER. Why isn't it available to the client connection? has anybody knew
> or written any middle layer for compression to deal with the WAN
> traffic? Any suggestion is appreciated.
>
> Thank you.
>
> -Vivien Kobayashi
>

Recap from the email,

>I don't understand. IConnect is client/server connections. ER/HDR is
>server-to-server.

What IBM was saying is if we make all servers local to the clients
which would improve the clients performance. I agree this might be
true, ex: instead of using 3Mbps bandwidth connecting to our server in
California, UK users will definitely see a lot of speed improvement if
the server is local to them with LAN speed(100/1000Mbps).


>Compression is useful mainly when the cost of compression is low
>compared to the saved cost of network transmission. We put it in for ER
>because with ER we are sending a lot of data to a lot of servers. Since
>the transmission is going to be large blocks (complete transactions) and
>we knew from testing that the savings would be significant, we added it
>for ER. We didn't think that we would get as much savings with HDR
>because we use logging compression and didn't think that we'd get good
>savings.

>For client server connections, the messages tend to be rather small, so
>the benefit would be even less.


Well, I think that depends on the return of the queries.. the main
table has a lot of columns (some clobs but mostly text, ideal for
compression) and the transmission between server and client is not
insignificant. We already know when the server and the clients are
connected via LAN, the performance is great; but not when the network
bandwidth is limited. I agree compression might be expensive for some;
however, it seems we have excessive power of server which is
shoe-shining most of time. So why not doing something useful, say
compression/decompression. Just a thought.

Thanks for your comments.

-Vivien Kobayashi

Madison Pruet wrote:
> Vivien Kobayashi wrote:
> > Is anyone aware of any way of compressing the data using IConnect or
> > any middleware?
> >
> >
> > Our main MFC developer has performed experiments by writing his own
> > middle layer with compression instead of connecting using IConnect.
> > These experiments show that IConnect is a major bottleneck. While the
> > native speed of the WAN links is indeed slower than the LAN, the
> > experiements got the WAN performance level up to an accetptable level.
> > However, this was only an experiment, and not pursued to develop a full
> > solution due to the complexity and cost of the implementation. We've
> > then talked to IBM/Informix but it did not go anywhere. The primary
> > suggestion we got is to move from HDR to ER but the complication and
> > the cost of ER is not preferrable at this point.
>
> I don't understand. IConnect is client/server connections. ER/HDR is
> server-to-server.
>
> Compression is useful mainly when the cost of compression is low
> compared to the saved cost of network transmission. We put it in for ER
> because with ER we are sending a lot of data to a lot of servers. Since
> the transmission is going to be large blocks (complete transactions) and
> we knew from testing that the savings would be significant, we added it
> for ER. We didn't think that we would get as much savings with HDR
> because we use logging compression and didn't think that we'd get good
> savings.
>
> For client server connections, the messages tend to be rather small, so
> the benefit would be even less.
>
> >
> > Our version of ideal solution is that the IConnect layer can be
> > optimized, potentially with built-in compression. However, it seems
> > compression is available but only among the servers participated in the
> > ER. Why isn't it available to the client connection? has anybody knew
> > or written any middle layer for compression to deal with the WAN
> > traffic? Any suggestion is appreciated.
> >
> > Thank you.
> >
> > -Vivien Kobayashi
> >

Vivien Kobayashi wrote:
> Is anyone aware of any way of compressing the data using IConnect or
> any middleware?
>
> In our current environment, we have quite a few clients connected to
> our 9.4 IDS via WAN.
> The windows GUI client application is written in MFC which uses a DLL
> that connects to the server using ESQL/C. The ESQL/C uses IConnect
> version 2.81tc3.
>
> The performance on the server is sufficient (CPU is usually 90% idle).
> The performance for the LAN clients is sufficient; however, the
> performance for the remote clients is not.
>
> Our main MFC developer has performed experiments by writing his own
> middle layer with compression instead of connecting using IConnect.
> These experiments show that IConnect is a major bottleneck. While the
> native speed of the WAN links is indeed slower than the LAN, the
> experiements got the WAN performance level up to an accetptable level.
> However, this was only an experiment, and not pursued to develop a full
> solution due to the complexity and cost of the implementation. We've
> then talked to IBM/Informix but it did not go anywhere. The primary
> suggestion we got is to move from HDR to ER but the complication and
> the cost of ER is not preferrable at this point.
>
> Our version of ideal solution is that the IConnect layer can be
> optimized, potentially with built-in compression. However, it seems
> compression is available but only among the servers participated in the
> ER. Why isn't it available to the client connection? has anybody knew
> or written any middle layer for compression to deal with the WAN
> traffic? Any suggestion is appreciated.
>
> Thank you.
>
> -Vivien Kobayashi
>

Are you sure your application is doing the right thing?
An example: Zip Code and it's description. If your application gets all the Zip-description table, waiting for the user to fill in the Zip Code to show the description it can lead to heavy data exchange between the client and the server... If it makes the query after getting the zip code from the user, it may have a slight delay, but the data exchanged will be much less.

The question you should ask is: The data that the user sees on screen justifies the slowness of the application? Did they use local or remote cursors...? etc.
I don't think you've posted the bandwidth of the WAN links, but compare that with the visible data on the application screen and decide if it's reasonable...

I recall a case where a customer was using MS environments, and there were some options (local/remote cursors if I recall correctly) that could make a huge difference...


Regards.

--
Fernando Nunes
Portugal

http://informix-technology.blogspot.com
My email works... but I don't check it frequently...

Starship Solutions has a product that does a secure DR for Informix
and I know they where looking at adding or had added compression to
speed transfer. I'm not aware of any tool that does the server to
server compression.



Eric B. Rowell


On 9/28/06, Fernando Nunes <spam@domus.online.pt> wrote:
> Vivien Kobayashi wrote:
> > Is anyone aware of any way of compressing the data using IConnect or
> > any middleware?
> >
> > In our current environment, we have quite a few clients connected to
> > our 9.4 IDS via WAN.
> > The windows GUI client application is written in MFC which uses a DLL
> > that connects to the server using ESQL/C. The ESQL/C uses IConnect
> > version 2.81tc3.
> >
> > The performance on the server is sufficient (CPU is usually 90% idle).
> > The performance for the LAN clients is sufficient; however, the
> > performance for the remote clients is not.
> >
> > Our main MFC developer has performed experiments by writing his own
> > middle layer with compression instead of connecting using IConnect.
> > These experiments show that IConnect is a major bottleneck. While the
> > native speed of the WAN links is indeed slower than the LAN, the
> > experiements got the WAN performance level up to an accetptable level.
> > However, this was only an experiment, and not pursued to develop a full
> > solution due to the complexity and cost of the implementation. We've
> > then talked to IBM/Informix but it did not go anywhere. The primary
> > suggestion we got is to move from HDR to ER but the complication and
> > the cost of ER is not preferrable at this point.
> >
> > Our version of ideal solution is that the IConnect layer can be
> > optimized, potentially with built-in compression. However, it seems
> > compression is available but only among the servers participated in the
> > ER. Why isn't it available to the client connection? has anybody knew
> > or written any middle layer for compression to deal with the WAN
> > traffic? Any suggestion is appreciated.
> >
> > Thank you.
> >
> > -Vivien Kobayashi
> >
>
> Are you sure your application is doing the right thing?
> An example: Zip Code and it's description. If your application gets all the Zip-description table, waiting for the user to fill in the Zip Code to show the description it can lead to heavy data exchange between the client and the server... If it makes the query after getting the zip code from the user, it may have a slight delay, but the data exchanged will be much less.
>
> The question you should ask is: The data that the user sees on screen justifies the slowness of the application? Did they use local or remote cursors...? etc.
> I don't think you've posted the bandwidth of the WAN links, but compare that with the visible data on the application screen and decide if it's reasonable...
>
> I recall a case where a customer was using MS environments, and there were some options (local/remote cursors if I recall correctly) that could make a huge difference...
>
>
> Regards.
>
> --
> Fernando Nunes
> Portugal
>
> http://informix-technology.blogspot.com
> My email works... but I don't check it frequently...
> _______________________________________________
> Informix-list mailing list
> Informix-list@iiug.org
> http://www.iiug.org/mailman/listinfo/informix-list
>